Final  Technical  Report  (A013) 

SERC  Technical  Task  Order  TOOOI 

Early  Identification  of  SE-Related  Program  Risks 

Opportunities  for  DoD  Systems  Engineering  (SE)  Transformation 
via  SE  Effectiveness  Measures  (EMs)  and  Evidence-Based  Reviews 


Barry  Boehm,  Dan  Ingold,  Winsor  Brown,  JoAnn  Lane,  George  Friedman  University  of  Southern  California 

Kathleen  Dangle,  Linda  Esker,  Forrest  Shull  Fraunhofer-Maryland 

Rich  Turner,  Jon  Wade,  Mark  Weitekamp  Stevens  Institute  of  Technology 
Paul  Componation,  Sue  O’Brien,  Dawn  Sabados,  Julie  Fortune  University  of  Alabama-Huntsville 


SYSTEMS  ENGINEERING 

Research  Center 


Contract  Number:  H98230-08-D-0171 
September  30,  2009 


Technical  Editor:  Alfred  Brown,  USC 


Report  Documentation  Page 


Form  Approved 
OMB  No.  0704-0188 


Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 


1.  REPORT  DATE 

30  SEP  2009 


2.  REPORT  TYPE 


3.  DATES  COVERED 


00-00-2009  to  00-00-2009 


4.  TITLE  AND  SUBTITLE 


Early  Identification  of  SE-Related  Program  Risks:  Opportunities  for 
DoD  Systems  Engineering  (SE)  Transformation  via  SE  Effectiveness 
Measures  (EMs)  and  Evidence-Based  Reviews 


5a.  CONTRACT  NUMBER 


5b.  GRANT  NUMBER 


5c.  PROGRAM  ELEMENT  NUMBER 


5d.  PROJECT  NUMBER 


5e.  TASK  NUMBER 


5f.  WORK  UNIT  NUMBER 


6.  AUTHOR(S) 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES)  8.  PERFORMING  ORGANIZATION 

Systems  Engineering  Research  Center, Stevens  Institute  of  Technology, 1  report  number 

Castle  Point  On  Hudson, Hoboken, NJ, 07030 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES)  10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

14.  ABSTRACT 

DoD  programs  need  effective  systems  engineering  (SE)  to  succeed.  DoD  program  managers  need  early 
warning  of  any  risks  to  achieving  effective  SE.  This  SERC  project  has  synthesized  analyses  of  DoD  SE 
effectiveness  risk  sources  into  a  lean  framework  and  toolset  for  early  identification  of  SE-related  program 
risks.  Three  important  points  need  to  be  made  about  these  risks.  ?  They  are  generally  not  indicators  of 
"bad  SE."  Although  SE  can  be  done  badly,  more  often  the  risks  are  consequences  of  inadequate  program 
funding  (SE  is  the  first  victim  of  an  underbudgeted  program),  of  misguided  contract  provisions  (when  a 
program  manager  is  faced  with  the  choice  between  allocating  limited  SE  resources  toward  producing 
contract-incentivized  functional  specifications  vs.  addressing  key  performance  parameter  risks,  the  path  of 
least  resistance  is  to  obey  the  contract),  or  of  management  temptations  to  show  early  progress  on  the  easy 
parts  while  deferring  the  hard  parts  till  later.  ?  Analyses  have  shown  that  unaddressed  risk  generally  leads 
to  serious  budget  and  schedule  overruns.  ?  Risks  are  not  necessarily  bad.  If  an  early  capability  is  needed, 
and  the  risky  solution  has  been  shown  to  be  superior  to  the  alternatives,  accepting  and  focusing  on 
mitigating  the  risk  is  generally  better  than  waiting  for  a  better  alternative  to  show  up.  Unlike  traditional 
schedule-based  and  event-based  reviews,  the  SERC  SE  EM  technology  enables  sponsors  and  performers  to 
agree  on  the  nature  and  use  of  more  effective  evidence-based  reviews.  These  enable  early  detection  of 
missing  SE  capabilities  or  personnel  competencies  with  respect  to  a  framework  of  Goals,  Critical  Success 
Factors  (CSFs),  and  Questions  determined  by  the  EM  task  from  the  leading  DoD  early-SE  CSF  analyses. 
The  EM  tools  enable  risk-based  prioritization  of  corrective  actions,  as  shortfalls  in  evidence  for  each 
question  are  early  uncertainties,  which  when  combined  with  the  relative  system  impact  of  a  negative 
answer  to  the  question,  translates  into  the  degree  of  risk  that  needs  to  be  managed  to  avoid  system 
overruns  and  incomplete  deliveries. 


15.  SUBJECT  TERMS 


16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 

18.  NUMBER 

19a.  NAME  OF 

ABSTRACT 

OF  PAGES 

RESPONSIBLE  PERSON 

a.  REPORT 

unclassified 

b.  ABSTRACT 

unclassified 

c.  THIS  PAGE 

unclassified 

Same  as 
Report  (SAR) 

140 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


TABLE  OF  CONTENTS 


I.  ABSTRACT  AND  SUMMARY . 1 

II.  INTRODUCTION . 1 

III.  METHODS,  ASSUMPTIONS,  AND  PROCEDURES . 3 

A.  Motivation  and  Context . 3 

B.  Technical  Approach . 3 

IV.  RESULTS  AND  DISCUSSION . 4 

A.  DoD  MDAP  SE  EM  Needs,  Opportunities,  and  Business  Case . 5 

1.  The  Need  and  Opportunity . 5 

2.  Evidence  Shortfalls  in  Current  SE  Practices . 6 

2.1  Technical  Shortfalls . 6 

2.2  Management  Shortfalls . 7 

3.  Consequences  of  Evidence  Shortfalls . 8 

B.  DoD  MDAP  SE  EM  Framework  and  Tools:  SEPRT  and  SECRT . 9 

1.  SEPRT  and  SECRT  Goal-Critical  Success  Factor-Question  Framework . 10 

2.  Question  Impact/Evidence  Ratings  and  Project  SE  Risk  Assessment . 13 

3.  Prototype  SE  Effectiveness  Risk  Tools . 14 

3.1  SE  Performance  Risk  Tool . 14 

3.2  SE  Competency  Risk  Tool . 17 

4.  Description  and  Usage  of  Prototype  Tools . 18 

4.1  Tool  Overview . 18 

4.2  Tool  Tailoring  and  Extension . 20 

5.  Summary  of  Framework  and  Tool  Evaluations . 21 

C.  DoD  MDAP  SE  EM  Concepts  of  Operation . 23 


23 


1.  Evidence  Criteria  and  Review  Milestone  Usage . 

2.  Use  of  the  SERC  SE  EM  Capabilities  in  Evidence-Based  SE . 24 

2.1  Scenario  1 :  MDA  Review  of  Acquirer  Plans  at  Milestone  A . 25 

2.2  Scenario  2:  Acquirer  and  Winning-Prototype  Developer  Contract  Negotiation . 25 

3.  FED  Development  Process  Framework . 27 

1 .  Commitment  Review  Process  Overview . 29 

4. 1  Examples  of  Successful  Experiences  with  Evidence-Based  Reviews . 29 

D.  Pilot  SEPRT  and  SECRT  Evaluation  Processes  and  Lessons  Learned . 30 

1.  Introduction . 30 

2.  Methodology . 30 

3.  Results . 31 

4.  Lessons  Learned . 32 

E.  SADB-DAPS  Based  SEPRT  and  SECRT  Evaluation  Summary . 32 

1.  SEPRT  and  DAPS/SADB  Study  Objectives  and  Approach . 32 

2.  Existing  Defense  Acquisition  Program  Support  Tools . 33 

3.  SEPRT  -  DAPS  Framework  Mapping  Results . 34 

4.  SADB  Analysis  Results . 37 

5.  Recommendations  and  Conclusions . 39 

F.  Conclusions  and  Recommendations . 40 

1.  Conclusions . 40 

2.  Recommendations . 41 

G.  References . 42 

H.  List  of  Acronyms . 44 

APPENDIX  A:  GOALS,  CRITICAL  SUCCESS  FACTORS,  AND  QUESTIONS . 47 


APPENDIX  B:  SEPRT  EXAMPLE  -  LOGISTICS  SUPPORT  SYSTEM  PROJECT 


61 


APPENDIX  C:  SECRT  EXAMPLE  -  LOGISTICS  SUPPORT  SYSTEM  PROJECT 


70 


APPENDIX  D:  BASIC  COVERAGE  MATRIX . 78 

APPENDIX  E:  SEPRT-DAPS  MAPPING . 83 

APPENDIX  F:  SERC  SE  EM:  PROPOSED  NEW  FRAMEWORK . 119 

APPENDIX  G:  EVOLUTION  OF  THE  EM  PROJECT  PLANS  AND  SCHEDULES . 122 

1.  Sponsor  Guidance . 122 

2.  Initial  Key  Activities . 123 

3.  Project  Schedule . 125 


APPENDIX  H.  COMPARISON  OF  SECRT  AND  DAU  SPRDE-SE/PSE  COMPETENCY  MODEL 
. 127 


TABLE  OF  FIGURES 


Figure  1.  Analysis  of  Negative  Findings  of  DAPS  Reviews . 7 

Figure  2.  Architecture  Evidence  Shortfall  Penalties  and  Resulting  Architecting  Investment  Sweet  Spots  8 

Figure  3.  Example  of  SEPRT  prototype . 15 

Figure  4.  Impact/risk  mapping  to  risk  exposure . 17 

Figure  5.  Detail  of  impact  and  evidence  rating . 19 

Figure  6.  Detail  of  risk  exposure  rollup  by  CSF . 20 

Figure  7.  Detail  of  risk  exposure  mapping . 20 

Figure  8.  Project  SysE  EM  Operational  Concept  (for  each  stage  of  system  definition  and  development) 


. 27 

Figure  9.  Overview  of  Commitment  Review  Process . 29 

Figure  10.  SEPRT  -  DAPS  Mapping  Sample . 35 

Figure  11.  Data  Set . 38 


TABLE  OF  TABLES 


Table  1.  Summary  of  Major  Scope  Decisions . 2 

Table  2.  Analysis  of  U.S.  Defense  Dept.  Major  Defense  Acquisition  Program  Portfolios . 6 

Table  3.  Dimensions  and  timescales  of  EM  evaluation . 10 

Table  4.  Review  of  candidate  EMs  by  research  teams . 11 

Table  5.  Goals  and  CSFs  for  SE  performance . 11 

Table  6.  Additional  goals  and  CSFs  for  SE  competency . 13 

Table  7.  Risk  impact  ratings . 15 

Table  8.  Risk  probability/evidence  ratings . 16 

Table  9.  Average  risk  exposure  calculation . 16 

Table  10.  Steps  for  Developing  a  FED . 28 

Table  11.  Revised  EM  Evaluation  Assignments . 124 

Table  12.  EM  Task  Schedule  and  Results . 125 


I.  ABSTRACT  AND  SUMMARY 


DoD  programs  need  effective  systems  engineering  (SE)  to  succeed. 

DoD  program  managers  need  early  warning  of  any  risks  to  achieving  effective  SE. 

This  SERC  project  has  synthesized  analyses  of  DoD  SE  effectiveness  risk  sources  into  a  lean  framework 
and  toolset  for  early  identification  of  SE-related  program  risks. 

Three  important  points  need  to  be  made  about  these  risks. 

•  They  are  generally  not  indicators  of  "bad  SE."  Although  SE  can  be  done  badly,  more  often  the  risks 
are  consequences  of  inadequate  program  funding  (SE  is  the  first  victim  of  an  underbudgeted 
program),  of  misguided  contract  provisions  (when  a  program  manager  is  faced  with  the  choice 
between  allocating  limited  SE  resources  toward  producing  contract-incentivized  functional 
specifications  vs.  addressing  key  performance  parameter  risks,  the  path  of  least  resistance  is  to  obey 
the  contract),  or  of  management  temptations  to  show  early  progress  on  the  easy  parts  while 
deferring  the  hard  parts  till  later. 

•  Analyses  have  shown  that  unaddressed  risk  generally  leads  to  serious  budget  and  schedule 
overruns. 

•  Risks  are  not  necessarily  bad.  If  an  early  capability  is  needed,  and  the  risky  solution  has  been 
shown  to  be  superior  to  the  alternatives,  accepting  and  focusing  on  mitigating  the  risk  is  generally 
better  than  waiting  for  a  better  alternative  to  show  up. 

Unlike  traditional  schedule-based  and  event-based  reviews,  the  SERC  SE  EM  technology  enables 
sponsors  and  performers  to  agree  on  the  nature  and  use  of  more  effective  evidence-based  reviews. 
These  enable  early  detection  of  missing  SE  capabilities  or  personnel  competencies  with  respect  to  a 
framework  of  Goals,  Critical  Success  Factors  (CSFs),  and  Questions  determined  by  the  EM  task  from 
the  leading  DoD  early-SE  CSF  analyses.  The  EM  tools  enable  risk-based  prioritization  of  corrective 
actions,  as  shortfalls  in  evidence  for  each  question  are  early  uncertainties,  which  when  combined  with 
the  relative  system  impact  of  a  negative  answer  to  the  question,  translates  into  the  degree  of  risk  that 
needs  to  be  managed  to  avoid  system  overruns  and  incomplete  deliveries. 


II.  INTRODUCTION 

The  mission  of  the  DoD  Systems  Engineering  Research  Center  (SERC)  is  to  perform  research  leading  to 
transformational  SE  methods,  processes,  and  tools  (MPTs)  that  enable  DoD  and  Intelligence 
Community  (IC)  systems  to  achieve  significantly  improved  mission  successes.  In  working  with  the 
DoD  EM  task  sponsors,  the  EM  team  converged  on  a  scope  and  direction  of  the  research  that  has 
created  and  piloted  a  set  of  EM  MPTs  that  has  strong  prospects  for  transforming  the  SE  process  into  an 
evidence-based  approach  that  enables  early  identification  and  resolution  of  program  risks. 
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The  major  scope  decisions  worked  out  with  the  sponsors  and  their  rationales  are  summarized  in  Table  1. 
For  example,  the  original  Statement  of  Work  specified  coverage  of  multiple  system  types:  weapons 
platforms,  systems  of  systems,  and  net-centric  services;  but  converged  on  MDAPs,  as  these  were  the 
DoD  programs  most  likely  to  benefit  from  improved  SE  EMs.  Similarly,  it  focused  on  SE  EMs 
common  to  ground,  sea,  air,  and  space  systems  rather  than  trying  to  cover  them  all.  However,  the  EMs 
and  tools  are  designed  to  be  easy  to  extend  to  special  domains  by  special-domain  organizations.  The 
development  of  prototype  tools  was  performed  both  to  facilitate  pilot  testing  of  the  EM  framework  and 
questions,  but  also  to  provide  an  early  demonstration  that  the  SERC  research  could  lead  to  the 
development  of  useful  capabilities.  Initially  some  sponsors  were  interested  in  research  on  the  return  on 
investment  for  specific  SE  practices,  but  we  found  insufficient  data  to  support  such  analyses,  and  instead 
have  structured  the  tools  so  that  they  could  serve  as  ways  to  generate  such  data  for  the  future.  More 
detail  on  the  evolution  of  the  EM  project’s  plans  and  schedules  is  provided  in  Appendix  G. 

Table  1.  Summary  of  Major  Scope  Decisions 


Decision 

MDAP  vs.  multi-type  EMs 
Core  vs.  all-domain  EMs 
Ease  of  tailoring,  extension 
Cover  SE  functional  performance 
and  personnel  competency 
Rate  both  degree  of  impact  and 
degree  of  satisfaction  evidence 
Hierarchical  goal  -  critical  success 
factor  -  question  framework 
Compatibility  with  INCOSE 
Leading  Indicators 
Framework  and  tools 
Pilot  use  and  evaluation 
Initial  focus  on  project 
assessment  vs.  practice  ROIs 
Initial  focus  on  early  SE 


Rationale 

SE  shortfalls  a  major  MDAP  problem 
Avoid  numerous  inapplicable  EMs 
Enable  special-community  tailoring 
Sponsor  priority 

Relation  to  risk  exposure  RE=P(L)*S(L),  ease 
of  tailoring  out  zero-impact  questions 
Ease  of  use,  understanding;  compatibility 
with  related  frameworks 
Complementary  coverage:  continuous  vs. 
discrete;  quantitative  vs.  qualitative 
Early  SERC  tangible  product 
Evidence  of  strengths  and  shortfalls 
ROI  data  unavailable;  could  be  generated 
via  tool  use 

Highest  leverage  on  outcomes _ 


Section  III  summarizes  the  overall  methods,  assumptions,  and  procedures  used  in  the  EM  task.  Section 
IV  begins  with  an  analysis  of  the  DoD  needs,  opportunities,  and  business  case  for  the  use  of  SE  EM 
methods,  process,  and  tools  (MPTs).  It  finds  that  the  business  case  is  very  strong  for  large  MDAPs,  and 
not  very  strong  for  ad-hoc,  quick-response  system  development.  It  then  elaborates  the  detail  of  the  SE 
EM  framework  and  tools,  the  concepts  of  operation  for  their  use,  their  use  on  pilot  projects,  and  their 
analysis  in  comparison  with  the  Defense  Acquisition  Program  Support  (DAPS)  Methodology  and  its 
Systemic  Analysis  Database  (SADB)  of  DAPS  assessment  results.  It  concludes  that  the  evidence  of 
utility  and  business-case  return  on  investment  is  sufficient  to  proceed  toward  enabling  their  broad  DoD 
use,  subject  to  performing  further  research  on  the  factors  necessary  to  ensure  the  EM  MPTs’  scalability, 
extendability,  and  adaptability  to  change,  and  to  performing  the  resulting  improvements  to  the  EM 
MPTs.  It  recommends  a  two-phase  approach  for  achieving  an  initial  operational  capability  and 
transitioning  it  to  a  sustaining  organization.  This  would  begin  with  research  on  approaches  to  the  key 
issues,  and  continue  with  incremental  elaboration,  experimentation,  and  refinement  of  the  preferred 
approaches. 
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III.  METHODS,  ASSUMPTIONS,  AND  PROCEDURES 
A.  Motivation  and  Context 

Numerous  General  Accountability  Office  reports,  Defense  Science  Board  and  National  Research 
Council  studies,  and  other  studies  have  addressed  the  magnitude  and  frequency  of  DoD  MDAP  budget 
and  schedule  overruns  and  delivery  deficiencies.  Many  of  these  have  identified  inadequate  SE  as  a 
major  source  of  the  problems.  Further  analyses  have  found  that  these  SE  inadequacies  have  largely 
resulted  from  commitments  to  proceed  based  on  inadequate  evidence  that  the  proposed  system  solutions 
can  actually  meet  DoD  needs  within  DoD’s  operational  environment  and  within  the  program’s  available 
budget  and  schedule  resources. 

This  research  project  was  commissioned  by  the  SERC  sponsors  to  identify  and  organize  a  framework  of 
SE  effectiveness  measures  (EMs)  that  could  be  used  to  assess  the  evidence  that  a  MDAP’s  SE  approach, 
current  results,  and  personnel  competencies  were  sufficiently  strong  to  enable  program  success. 
Another  component  of  the  research  was  to  formulate  operational  concepts  that  would  enable  MDAP 
sponsors  and  performers  to  use  the  EMs  as  the  basis  of  collaborative  formulation,  scoping,  planning,  and 
monitoring  of  the  program’s  SE  activities,  and  to  use  the  monitoring  results  to  steer  the  program  toward 
the  achievement  of  feasible  SE  solutions. 


B.  Technical  Approach 

The  EM  research  project  reviewed  over  two-dozen  sources  of  candidate  SE  EMs,  and  converged  on  the 
strongest  sources  to  be  used  to  identify  candidate  SE  EMs.  We  developed  a  coverage  matrix  to 
determine  the  envelope  of  candidate  EMs,  and  the  strength  of  consensus  on  each  candidate  EM.  It  fed 
the  results  back  to  the  source  originators  to  validate  the  coverage  matrix  results.  This  resulted  in  further 
insights  and  added  candidate  EMs  to  be  incorporated  into  an  SE  Performance  Risk  Framework.  The 
resulting  framework  is  organized  into  a  hierarchy  with  4  Goals,  18  Critical  Success  Factors,  and  74 
Questions  that  appeared  to  cover  the  central  core  of  common  SE  performance  determinants  of  SE 
effectiveness. 

Concurrently,  the  research  project  was  extended  to  also  assess  SE  personnel  competency  as  a 
determinant  of  program  success.  We  analyzed  an  additional  six  personnel  competency  risk  frameworks 
and  sets  of  questions.  Their  Goals  and  Critical  Success  Factors  were  very  similar  to  those  used  in  the 
SE  Performance  Risk  Framework,  although  the  Questions  were  different.  The  resulting  SE  Competency 
Risk  Framework  added  one  further  Goal  of  Professional  and  Interpersonal  Skills  with  five  Critical 
Success  Factors,  resulting  in  a  framework  of  5  Goals,  23  Critical  Success  Factors,  and  81  Questions. 

In  order  to  evaluate  the  SE  Performance  and  Competency  Risk  Frameworks,  we  developed  simple,  easy- 
to-use  spreadsheet  tools  for  pilot  projects  to  use  and  provide  feedback  on  the  utility  of  the  frameworks 
and  tools.  The  tools  are  called  the  SE  Performance  Risk  Tool  (SEPRT)  and  the  SE  Competency  Risk 
Tool  (SECRT).  The  initial  round  of  7  pilot  projects  yielded  quite  positive  overall  assessment  result,  as 
discussed  in  Sections  IV.B.5  and  IY.D.  It  also  provided  several  valuable  suggestions  for  improvement, 
some  of  which  have  already  been  implemented. 

We  performed  a  complementary  analysis  comparing  the  coverage  of  the  SE  EMs  with  the  content  of  the 
DAPS  Methodology,  and  with  respect  to  the  initial  results  of  queries  to  the  Systemic  Analysis  Database 
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(SADB)  of  negative  findings  resulting  from  DAPS  assessments.  The  results  again  were  largely  positive, 
as  discussed  in  Section  IY.E.  Overall,  the  DAPS  Methodology  goes  into  greater  detail  in  its  questions, 
providing  a  complementary  capability  for  users  of  the  SEPRT  and  SECRT  to  apply  in  focusing  in  on 
their  high-impact  questions. 

We  developed  operational  concepts  for  the  use  of  the  SEPRT  and  SECRT  on  MDAPs  for  three  early  life 
cycle  stages.  Each  involves  collaborative  efforts  by  the  program  sponsors  and  performers.  For  each 
question,  the  sponsors  rate  the  relative  impact  on  the  program  if  the  answer  is  negative,  and  a  scale  of 
Critical  (40-100%  overrun  or  its  equivalent  in  delivery  shortfalls);  Significant  (20-40%  overrun); 
Moderate  (2-20%  overrun);  and  Little  or  No  Impact  (0-2%  overrun).  These  ratings  are  provided  to  the 
performers,  who  identify  situations  in  which  the  ratings  appear  inconsistent  with  the  program’s 
objectives,  available  resources,  contract  provisions,  or  likely  system  impact.  These  situations  stimulate 
constructive  discussion  and  a  more  compatible  shared  vision  and  feasible  set  of  impact  ratings  and 
program  parameters  to  be  used  in  managing  the  program. 

Once  consensus  is  reached  on  the  impact  ratings,  the  performer  SEs  proceed  to  define  and  develop  the 
system,  along  with  evidence  that  the  questions  are  being  satisfactorily  addressed.  The  evidence  is 
evaluated  by  independent  experts  at  each  major  review.  Shortfalls  in  evidence  are  identified  as 
uncertainties  and  risk  probabilities,  which  when  combined  with  the  question’s  risk  impact  determine  the 
level  of  risk  exposure  associated  with  the  question.  To  reinforce  the  importance  of  the  evidence-based 
assessments,  a  considerable  portion  of  the  performer’s  award  fee  is  based  on  the  degree  to  which  the 
performer  evidence  has  shown  the  project  to  be  at  a  low  level  of  risk. 

The  evidence  is  thus  a  first-class  deliverable,  as  compared  to  being  an  optional  appendix  in  most  current 
programs,  whose  content  is  largely  dropped  as  project  resources  become  scarce.  As  such,  its 
development  needs  to  be  planned,  budgeted,  and  made  a  key  element  in  the  project’s  earned  value 
management  system.  Any  risks  resulting  from  shortfalls  in  evidence  need  to  be  addressed  by  risk 
mitigation  plans  and  their  associated  resource  requirements.  Both  evidence  generation  and  risk 
mitigation  add  up-front  effort,  but  a  business  case  analysis  of  the  effects  of  going  from  minimal  to 
thorough  architecture  and  risk  resolution  on  161  software-intensive  systems  yields  significant  returns  on 
investment  for  MDAP-scale  projects,  as  discussed  in  Section  IV.A.3. 


IV.  RESULTS  AND  DISCUSSION 

This  section  begins  with  a  summary  of  the  DoD  needs  for  improvements  in  overall  systems  acquisition, 
and  particularly  in  systems  engineering.  One  of  the  particular  needs  in  DoD  systems  engineering  is  for 
better  ways  for  measuring  its  effectiveness,  as  it  is  difficult  to  manage  something  that  one  is  not  able  to 
measure.  A  particular  opportunity  is  for  SE  EMs  that  focus  on  measurable  and  independently  verifiable 
evidence  of  effectiveness,  and  associated  evidence-based  reviews  that  enable  MDAPs  to  identify 
shortfalls  in  evidence  of  the  feasibility  of  the  specifications  and  plans  being  presented  for  approval  at 
milestone  reviews.  This  opportunity  led  us  to  associate  the  SE  EMs  with  operational  concepts 
highlighting  their  use  in  evidence-based  reviews.  Section  A  also  presents  a  business  case  for  the 
thoroughness  of  SE  activities,  which  shows  that  the  payoff  for  evidence-based  SE  EMs  is  greatest  for 
large  MDAPs. 

Section  B  proceeds  to  describe  the  derivation  of  the  SE  EM  frameworks  of  Goals,  Critical  Success 
Factors,  and  Questions  for  performance  and  personnel  competency  from  leading  DoD  studies.  It  then 
describes  the  Systems  Engineering  Performance  Risk  Tool  (SEPRT)  and  Systems  Engineering 
Competency  Risk  Tool  (SECRT)  tools  developed  to  support  pilot  evaluation  of  the  SE  EM  frameworks, 
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and  then  summarizes  the  results  of  the  pilot  studies.  Section  C  then  presents  several  operational 
concepts  for  using  the  tools,  both  at  major  DoD  milestones  and  during  planning  and  execution  of  DoD 
MDAP  projects.  Section  D  provides  a  more  detailed  summary  of  two  of  the  pilot  evaluations,  and 
Section  E  summarizes  a  complementary  evaluation  of  the  framework  via  its  comparison  to  the  DoD 
Defense  Acquisition  Program  Support  (DAPS)  methodology  and  its  associated  Systemic  Analysis 
Database  (SADB)  of  DAPS  program  assessment  results. 

Section  F  summarizes  the  conclusions  that  the  business  case,  the  pilot  results,  and  the  DAPS  and  SADB 
comparison  results  all  point  to  a  strong  payoff  for  DoD  use  of  the  SE  SM  framework,  tools,  and 
evidence-based  concepts  of  operation.  It  also  provides  recommendations  for  a  two-phase  approach  for 
achieving  an  initial  operational  capability  and  transitioning  it  to  a  sustaining  organization.  This  would 
begin  with  research  on  approaches  to  the  key  issues,  and  continue  with  incremental  elaboration, 
experimentation,  and  refinement  of  the  preferred  approaches. 


A.  DoD  MDAP  SE  EM  Needs,  Opportunities,  and  Business  Case 

1.  The  Need  and  Opportunity 

Table  2,  obtained  from  a  recent  General  Accountability  Office  (GAO)  report  [29],  shows  the  magnitude 
of  the  problems  to  be  addressed  by  the  SE  EM  capabilities.  These  included  total  DoD  annual  MDAP 
cost  growths  of  roughly  $300  billion  per  year  and  delivery  delays  coming  close  to  two  years.  In  many 
cases,  these  “cost  and  schedule  growths”  were  not  actual  growths,  but  the  results  of  traditional 
acquisition  practices  requiring  programs  to  associate  costs  and  schedules  with  capabilities  before 
evidence  of  technical,  cost,  and  schedule  feasibility  was  available. 
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Table  2.  Analysis  of  U.S.  Defense  Dept.  Major  Defense  Acquisition  Program  Portfolios 


Analysis  of  U.S.  Defense  Department  Major  Defense  Acquisition  Program  Portfolios 

Fiscal  2009  dollars 

Portfolio  size 

2003 

2007 

2008 

Number  of  programs 

77 

95 

96 

Total  planned  commitments 

$1.2  trillion 

$1.6  trillion 

$1.6  trillion 

Commitments  outstanding 

$724.2  billion 

$875.2  billion 

$786.3  billion 

Portfolio  indicators 

Change  to  total  RDT&E*  costs  from  first 
estimate 

37% 

40% 

42% 

Change  to  total  acquisition  costs  from  first 
estimate 

19% 

26% 

25% 

Total  acquisition  cost  growth 

$183  billion 

$301.3  billion 

$296.4  billion 

Share  of  programs  with  25%  increase 
in  program  acquisition  unit  cost  growth 

41% 

44% 

42% 

Average  schedule  delay  in  delivering  initial 
capabilities 

18  months 

21  months 

22  months 

Source:  U.S.  Government  Accountability  Office  ^Research,  Development,  Testing  &  Evaluation 

Providing  such  validated  evidence  is  generally  considered  to  be  a  good  practice,  but  generally  fails  to  be 
done  well.  This  is  because  of  a  lack  of  evidence  criteria;  a  lack  of  evidence-generation  procedures  and 
measures  for  monitoring  evidence  generation;  a  lack  of  appreciation  of  the  consequences  of  proceeding 
into  development  with  unsubstantiated  specifications  and  plans;  and  because  of  current  methods, 
standards,  and  contractual  provisions  that  make  evidence  generation  optional.  The  main  contributions 
of  the  SERC  SE  EM  task  are  to  provide  experience-based  approaches  for  each  of  these  concerns  and  to 
illustrate  the  consequences  of  their  use  via  case  studies  and  parametric  analysis. 

2.  Evidence  Shortfalls  in  Current  SE  Practices 

2.1  Technical  Shortfalls 

Current  system  design  and  development  methods  focus  strongly  on  the  inputs  and  outputs,  preconditions 
and  post-conditions  that  a  system  function,  component,  or  service  operates  by  as  a  product.  They  lack 
adequate  capabilities  to  support  evidence  about  how  well  the  elements  perform,  how  expensive  they  will 
be  to  develop,  or  how  compatible  are  their  underlying  assumptions.  In  principle,  they  support  reasoning 
about  off-nominal  performance,  but  in  practice  their  descriptions  generally  focus  on  sunny-day 
scenarios.  As  a  result,  many  DoD  MDAP  project  reviews  tend  to  focus  on  exhaustive  presentations  of 
PowerPoint  charts  and  Unified  Modeling  Language  (UML)  diagrams.  They  provide  little  evidence  that 
the  system  they  describe  could  handle  rainy-day  scenarios;  perform  adequately  on  throughput,  response 
time,  safety,  security,  usability,  or  other  desired  quality  attributes  across  a  range  of  representative 
mission  scenarios;  be  buildable  within  the  available  budgets  and  schedules  in  the  plan;  or  generate 
positive  returns  on  investment  for  the  stakeholders. 

Figure  1  shows  a  summary  from  a  2007  National  Defense  Industrial  Association  (NDIA)  workshop  of 
analyzing  the  underlying  causes  of  the  negative  findings  of  DAPS  reviews  of  several  dozen  DoD 
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programs.  It  shows  that  two  of  the  top  four  causes  of  negative  findings  were  due  to  SE  shortfalls  in 
technical  processes  and  requirements  processes,  shown  in  bold.  Several  others,  shown  in  italics,  were 
partly  due  to  inadequate  SE  performance,  often  caused  by  the  fact  that  adequate  SE  is  the  first  victim  of 
a  lack  of  program  realism  with  respect  to  the  necessary  budgets  and  schedules  needed  to  do  a  thorough 
job. 


SE  Performance,  Competency  are  Major  Sources  of 

OSD/AT&L  Systemic  Analysis  Negative  Findings 

1  Technical  process  (35  instances) 

6  Lack  of  appropriate  staff  (23) 

-  V&V,  integration,  modeling  &  sim. 

2  Management  process  (31) 

7  Ineffective  organization  (22) 

3  Acquisition  practices  (26) 

8  ineffective  communication  (21) 

4  Requirements  process  (25) 

9  Program  realism  (21) 

5  Competing  priorities  (23) 

1 0  Contract  structure  (20) 

Figure  1.  Analysis  of  Negative  Findings  of  DAPS  Reviews 


2.2  Management  Shortfalls 

A  major  recent  step  forward  in  the  management  of  outsourced  government  projects  has  been  to  move 
from  schedule-based  reviews  to  event-based  reviews.  A  schedule-based  review  says  basically  that:  “The 
contract  specifies  that  the  Preliminary  Design  Review  (PDR)  will  be  held  on  June  1,  2011,  whether  we 
have  a  design  or  not.” 

In  general,  neither  the  customer  nor  the  developer  wants  to  fail  the  PDR,  so  the  project  goes  into 
development  with  the  blessing  of  having  passed  a  PDR,  but  with  numerous  undefined  interfaces  and 
unresolved  risks.  As  shown  below,  these  will  generally  result  in  major  project  overruns  and  incomplete 
deliveries. 

An  event-based  review  says:  “Once  we  have  a  preliminary  design,  we  will  hold  the  PDR.”  Such  a 
review  will  generally  consist  of  exhaustive  presentations  of  sunny-day  PowerPoint  charts  and  UML 
diagrams.  This  is  largely  because  most  traditional  DoD  acquisition  contracts  and  procedures  have  an 
early  focus  on  product-oriented  versus  feasibility-evidence-oriented  deliverables  and  reviews. 

These  reinforce  paths  toward  project  disaster,  as  in  this  quote  from  a  recent  large -project  manager:  “I’d 
like  to  have  some  of  my  systems  engineers  address  those  software  quality-factor  risks,  but  my  contract 
deliverables  and  award  fees  are  based  on  having  all  of  the  system’s  functions  defined  by  the  next 
review.”  Similar  over-focus  on  product  definition  is  found  in  project  eamed-value  management  systems 
for  tracking  project  progress  and  Data  Item  Descriptions  (DIDs)  for  deliverables.  Most  contract  DIDs 
cover  function,  interface,  and  infrastructure  considerations,  but  place  demonstration  of  their  feasibility 
in  optional  appendices  where,  as  with  the  project  manager  above,  they  are  the  first  to  go  when  time  and 
effort  are  running  out. 
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3.  Consequences  of  Evidence  Shortfalls 


This  is  not  just  a  DoD  problem,  but  a  pervasive  problem  in  system  acquisition  and  outsourcing.  For 
example,  the  biannual  Standish  Reports  consistently  identify  shortfalls  in  evidence  of  feasibility  with 
respect  to  stakeholder  objectives  as  the  major  root  causes  of  project  failure.  The  2009  Standish  Report 
[29]  found  that  only  32%  of  the  9000  projects  reported  delivered  their  full  capability  within  their  budget 
and  schedule;  24%  were  cancelled;  and  44%  were  significantly  over  budget,  over  schedule,  and/or 
incompletely  delivered.  More  detail  on  the  top  critical  success  factors  distinguishing  successful  from 
failed  software  projects  was  in  the  2005  Standish  Report.  There,  71%  of  the  sources  of  failure  were 
primarily  due  to  evidence  shortfalls  with  respect  to  stakeholder  objectives  (lack  of  user  involvement, 
executive  support,  clear  requirements,  proper  planning,  and  realistic  expectations). 

Recent  further  analyses  of  the  Constructive  Cost  Model  (COCOMO)  II  database  on  the  effects  of 
incomplete  architecture  definition,  risk  resolution,  and  resulting  feasibility  evidence  on  software¬ 
intensive  systems  are  shown  in  Figure  2.  They  show  the  results  of  a  risk-driven  “how  much  architecting 
is  enough?”  analysis,  based  on  the  COCOMO  II  Architecture  and  Risk  Resolution  (RESL)  factor  [8], 
[13].  This  factor  was  calibrated  along  with  22  others  to  161  project  data  points.  It  relates  the  amount  of 
extra  rework  effort  on  a  project  to  the  degree  of  evidence  that  the  project  had  demonstrated  its 
architecture  feasibility  and  resolved  its  technical  and  management  risks.  This  also  correlates  with  the 
percent  of  project  effort  devoted  to  software-intensive  system  architecting  and  risk  resolution. 


Effects  of  Size  on  Sweet  Spots  Effects  of  Volatility  and  Criticality 


Figure  2.  Architecture  Evidence  Shortfall  Penalties  and  Resulting  Architecting  Investment  Sweet 

Spots 

The  analysis  indicated  that  the  amount  of  rework  was  an  exponential  function  of  project  size.  A  small 
(10  thousand  equivalent  source  lines  of  code,  or  KSLOC)  project  could  fairly  easily  adapt  its 
architecture  to  rapid  change  via  refactoring  or  its  equivalent,  with  a  rework  penalty  of  18%  between 
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minimal  and  extremely  thorough  architecture  and  risk  resolution.  However,  a  very  large  (10,000 
KSLOC)  project  would  incur  a  corresponding  rework  penalty  of  91%,  covering  such  effort  sources  as 
integration  rework  due  to  undiscovered  large-component  interface  incompatibilities,  technology 
immaturities,  and  critical  performance  shortfalls. 

The  effects  of  rapid  change  (volatility)  and  high  dependability  (criticality)  on  the  architecture  evidence 
shortfall  penalties  and  resulting  architecting  investment  sweet  spots  are  shown  in  the  right  hand  graph. 
Here,  the  solid  black  lines  represent  the  average-case  cost  of  rework,  architecting,  and  total  cost  for  a 
100-KSLOC  project  as  shown  at  the  left.  The  dotted  red  lines  show  the  effect  on  the  cost  of 
architecting  and  total  cost  if  rapid  change  adds  50%  to  the  cost  of  architecture  and  risk  resolution. 
Quantitatively,  this  moves  the  sweet  spot  from  roughly  20%  to  10%  of  effective  architecture  investment 
(but  actually  15%  due  to  the  50%  cost  penalty).  Thus,  high  investments  in  architecture,  feasibility 
analysis,  and  other  documentation  do  not  have  a  positive  return  on  investment  for  very  high-volatility 
projects  due  to  the  high  costs  of  documentation  rework  for  rapid-change  adaptation. 

The  dashed  blue  lines  at  the  right  represent  a  conservative  analysis  of  the  cost  effects  of  system  failure 
due  to  unidentified  architecting  shortfalls.  It  assumes  that  the  costs  of  architecting  shortfalls  are  not 
only  added  rework,  but  also  losses  to  the  organization’s  operational  effectiveness  and  productivity. 
These  are  conservatively  assumed  to  add  50%  to  the  project-rework  cost  of  architecture  shortfalls  to  the 
organization.  In  most  cases  for  high-assurance  systems,  the  added  cost  would  be  considerably  higher. 

Quantitatively,  this  moves  the  sweet  spot  from  roughly  20%  to  over  30%  as  the  most  cost-effective 
investment  in  architecting  and  development  of  feasibility  evidence  for  a  100-KSLOC  project.  It  is  good 
to  note  that  the  “sweet  spots”  are  actually  relatively  flat  “sweet  regions”  extending  5-10%  to  the  left  and 
right  of  the  “sweet  spots.”  However,  moving  to  the  edges  of  a  sweet  region  increases  the  risk  of 
significant  losses  if  some  project  assumptions  turn  out  to  be  optimistic. 

The  bottom  line  for  Figure  2  is  that  the  greater  the  project’s  size,  criticality,  and  stability  are,  the  greater 
is  the  need  for  validated  architecture  feasibility  evidence.  However,  for  very  small,  low-criticality 
projects  with  high  volatility,  the  evidence-producing  efforts  would  make  little  difference  and  would 
need  to  be  continuously  redone,  producing  a  negative  return  on  investment.  In  such  cases,  agile 
methods  such  as  rapid  prototyping,  Scrum  [27]  and  extreme  Programming  [3]  will  be  more  effective. 
Overall,  evidence-based  specifications  and  plans  are  a  much  better  match  to  MDAPs,  where  in  general 
they  will  eliminate  many  of  the  system  delivery  overruns  and  shortfalls  experienced  on  current  MDAPs. 


B.  DoD  MDAP  SE  EM  Framework  and  Tools:  SEPRT  and  SECRT 

Our  research  suggests  that  systems  engineering  (SE)  effectiveness  measures  (EMs)  can  be  characterized 
along  two  major  dimensions  of  risk,  and  across  two  timescales.  SE  effectiveness  can  be  assessed  both 
by  the  performance  of  the  SE  function,  and  by  the  competency  of  those  performing  it.  Each  dimension 
can  be  evaluated  at  discrete  decision  points  in  a  program,  and  also  in  a  continuous  fashion  throughout  its 
execution. 

We  propose  three  frameworks  to  evaluate  SE  EMs  along  these  dimensions  and  timescales,  and  have 
created  prototype  tools  to  evaluate  two  of  these  frameworks  in  the  discrete  dimension:  the  Systems 
Engineering  Performance  Risk  Tool  (SEPRT),  and  the  Systems  Engineering  Competency  Risk  Tool 
(SECRT).  Complementing  these  discrete-evaluation  frameworks,  research  by  INCOSE  on  continuous 
evaluation  has  led  to  conceptual  development  of  a  set  of  leading  indicators  (LI),  which  were  included  as 
candidates  in  developing  the  SEPRT  performance  evaluation  questions  (See  Table  3.) 


9 


Table  3.  Dimensions  and  timescales  of  EM  evaluation 


Discrete 

SEPRT 

SECRT 

Continuou 

s 

INCOSE  Leading  Indicators 

Performance 

Competency 

This  section  discusses  the  evolution  of  the  SE  EM  evaluation  frameworks,  and  the  specific 
implementations  of  these  frameworks  into  the  prototype  SEPRT  and  SECRT  tools.  We  also  present 
preliminary  results  of  pilot  evaluations  of  the  SEPRT  and  SECRT  tools,  which  were  developed  to  help 
determine  the  utility  of  the  proposed  EM  frameworks  across  multiple  acquisition  frameworks  and 
application  domains. 


1.  SEPRT  and  SECRT  Goal-Critical  Success  Factor-Question  Framework 

Our  initial  research  focused  on  identifying  methods  that  might  be  suitable  for  assessing  the  effectiveness 
of  systems  engineering  on  major  defense  acquisition  programs  (MDAPs).  A  literature  review  identified 
eight  candidate  measurement  methods,  as  follows: 

•  NRC  Pre -Milestone  A  &  Early-Phase  SysE  top-20  checklist  [20] 

•  Air  Force  Probability  of  Program  Success  (PoPS)  Framework  [1] 

•  INCOSE/LMCO/MIT  Leading  Indicators  [24] 

•  Stevens  Leading  Indicators  (new;  using  SADB  root  causes)  [34] 

•  USC  Anchor  Point  Feasibility  Evidence  progress  [31] 

•  UAH  teaming  theories  [14] 

•  NDIA/SEI  capability/challenge  criteria  [15] 

•  SISAIG  Early  Warning  Indicators  [9]  /  USC  Macro  Risk  Tool  [33] 

Pages  5-8  of  the  NRC  report  [20]  suggests  a  “Pre -Milestone  A/B  Checklist”  for  judging  the  successful 
completion  of  early-phase  systems  engineering.  Using  this  checklist  as  a  concise  starting  point,  the 
researchers  identified  similar  key  elements  in  each  of  the  other  candidate  measurement  methods, 
resulting  in  a  list  of  44  characteristics  of  effective  systems  engineering  (see  Appendix  C).  Multiple 
research  teams  independently  examined  two  or  more  candidate  EMs  to  assess  whether  and  to  what 
degree  each  characteristic  was  addressed  by  the  respective  measure,  as  noted  in  Table  4.  This 
assessment  also  identified  another  six  EM  characteristics  not  previously  noted. 
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Table  4.  Review  of  candidate  EMs  by  research  teams 


Candidate  EM 

USC 

Stevens 

FC-MD 

UAH 

PoPS  Leading  Indicators  (Us) 

X 

X 

X 

INCOSE  Us 

X 

X 

Stevens  Us 

X 

X 

X 

SISAIG  Us/  Macro  Risk 

X 

X 

X 

NRC  Top-20  List 

X 

X 

X 

SEI  CMMI-Based  Us 

X 

X 

X 

USC  AP-Feasibility  Evidence 

X 

X 

X 

UAH  Team  Effectiveness 

X 

X 

X 

Previous  research  by  the  USC  team  into  a  macro-risk  model  for  large-scale  projects  had  resulted  in  a 
taxonomy  of  high-level  goals  and  supporting  critical  success  factors  (CSFs)  based  on  [28].  This  was 
identified  as  a  potential  framework  for  organizing  the  5 1  EM  characteristics  identified  above.  Analysis 
of  the  characteristics  showed  that  they  could  be  similarly  organized  into  a  series  of  four  high-level  goals, 
each  containing  4-5  CSFs,  as  seen  in  Table  5.  Our  survey  of  the  existing  literature  suggests  that  these 
CSFs  are  among  the  factors  that  are  most  critical  to  successful  SE,  and  that  the  degree  to  which  the  SE 
function  in  a  program  satisfies  these  CSFs  is  a  measure  of  SE  effectiveness. 

Table  5.  Goals  and  CSFs  for  SE  performance 


High-level  Goals 

Critical  Success  Factors 

Concurrent 
definition  of 
system 

requirements  & 
solutions 

Understanding  of  stakeholder  needs 

Concurrent  exploration  of  solutions 

System  scoping  &  requirements 
definition 

Prioritization/allocation  of  requirements 

System  life-cycle 
organization, 
planning  &  staffing 

Establishment  of  stakeholder  RAAs 

Establishment  of  IPT  RAAs 

Establishment  of  resources  to  meet 
objectives 

Establishment  of 
selection/contracting/incentives 

Assurance  of  necessary  personnel 
competencies 

Technology 
maturing  & 
architecting 

COTS/NDI  evaluation,  selection, 
validation 

Life-cycle  architecture  definition  & 
validation 
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High-level  Goals 

Critical  Success  Factors 

Use  of  prototypes,  models,  etc.  to 
validate  maturity 

Validated  budgets  &  schedules 

Evidence-based 
progress 
monitoring  & 
commitment 
reviews 

Monitoring  of  system  definition 

Monitoring  of  feasibility  evidence 
development 

Monitoring/assessment/re-planning  for 
changes 

Identification  and  mitigation  for 
feasibility  risks 

Reviews  to  ensure  stakeholder 
commitment 

Related  to  the  effectiveness  measures  of  SE  performance  is  the  need  to  measure  the  effectiveness  of  the 
staff  assigned  to  the  SE  function.  Besides  the  eight  SEPRT  sources,  six  additional  sources  were 
reviewed  for  contributions  to  Personnel  Competency  evidence  questions.  These  were: 

•  Office  of  the  Director  of  National  Intelligence  (ODNI),  Subdirectory  Data  Collection  Tool:  Systems 
Engineering  [22] 

•  INCOSE  Systems  Engineering  Handbook,  August  2007  [17] 

•  ASN  (RD&A),  Guidebook  for  Acquisition  of  Naval  Software  Intensive  Systems,  September  2008 

[3] 

•  CMU/SEI,  Models  for  Evaluating  and  Improving  Architecture  Competence  [4] 

•  NASA  Office  of  the  Chief  Engineer,  NASA  Systems  Engineering  Behavior  Study,  October  2008 
[34] 

•  National  Research  Council,  Human-System  Integration  in  the  System  Development  Process,  2007 
[23] 

These  were  analyzed  for  candidate  knowledge,  skills,  and  abilities  (KSA)  attributes  proposed  for 
systems  engineers.  Organizing  these  work  activities  and  KSAs  revealed  that  the  first  four  goals  and 
their  CSFs  were  in  common  with  the  EM  taxonomy.  An  additional  goal  and  its  related  CSFs  were  also 
discovered,  as  presented  in  Table  6. 
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Table  6.  Additional  goals  and  CSFs  for  SE  competency 


High-level  Goal 

Critical  Success  Factors 

Professional  and 
interpersonal  skills 

Ability  to  plan,  staff,  organize,  team- 
build,  control,  and  direct  systems 
engineering  teams 

Ability  to  work  with  others  to  negotiate, 
plan,  execute,  and  coordinate 
complementary  tasks  for  achieving 
program  objectives 

Ability  to  perform  timely,  coherent,  and 
concise  verbal  and  written 
communication 

Ability  to  deliver  on  promises  and 
behave  ethically 

Ability  to  cope  with  uncertainty  and 
unexpected  developments,  and  to  seek 
help  and  fill  relevant  knowledge  gaps 

2.  Question  Impact/Evidence  Ratings  and  Project  SE  Risk  Assessment 

Using  these  relatively  high-level  criteria,  however,  it  is  difficult  to  evaluate  whether  the  SE  on  a 
particular  program  adequately  satisfies  the  CSFs.  In  its  approach  to  evaluating  macro-risk  in  a  program, 
[31]  suggests  that  a  goal-question-metric  (GQM)  approach.  [4]  provides  a  method  to  accomplish  this 
evaluation.  Following  this  example,  the  researchers  developed  questions  to  explore  each  goal  and  CSF, 
and  devised  metrics  to  determine  the  relevance  of  each  question  and  the  quality  of  each  answer. 

The  researchers  began  question  development  for  the  SE  performance  framework  with  the  checklist  from 
[20],  Further  questions  were  adapted  from  the  remaining  EM  characteristics,  rewritten  as  necessary  to 
express  them  in  the  form  of  a  question.  Each  question  is  phrased  such  that,  answered  affirmatively,  it 
indicates  positive  support  of  the  corresponding  CSF.  We  hypothesize  that  the  strength  of  support  for 
each  answer  is  related  to  the  relative  risk  probability  associated  with  the  CSF  that  question  explores. 

Rather  than  rely  simply  on  the  opinion  of  the  evaluator  as  to  the  strength  of  the  response,  a  more 
quantifiable  evidence-based  approach  was  selected.  The  strength  of  the  response  is  related  to  the 
amount  of  evidence  available  to  support  an  affirmative  answer — the  stronger  the  evidence,  the  lower  the 
risk  probability.  Feedback  from  industry,  government,  and  academic  participants  in  workshops 
conducted  in  March  and  May  2009  suggested  that  a  simple  risk  probability  scale  with  four  discrete 
values  be  employed  for  this  purpose. 

Evidence  takes  whatever  form  is  appropriate  for  the  particular  question.  For  example,  a  simulation 
model  might  provide  evidence  that  a  particular  performance  goal  can  be  met.  Further,  the  strongest 
evidence  is  that  which  independent  expert  evaluators  have  validated. 

Recognizing  that  each  characteristic  might  be  more  or  less  applicable  to  a  particular  program  being 
evaluated,  the  questions  are  also  weighted  according  to  the  risk  impact  that  failure  to  address  the 
question  might  be  expected  to  have  on  the  program.  Again  based  on  workshop  feedback,  a  four- value 
scale  for  impact  was  chosen. 
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The  product  of  the  magnitude  of  a  potential  loss  (the  risk  impact)  and  the  likelihood  of  that  loss  (the  risk 
probability)  is  the  risk  exposure.  Although  risk  exposure  is  generally  calculated  given  quantitative  real- 
number  estimates  of  the  magnitude  and  probabilities  of  a  loss,  the  assessments  of  risk  impact  and  risk 
probability  described  above  use  an  ordinal  scale.  Therefore,  we  employ  a  mapping  between  the  four- 
value  risk  probability  and  risk  impact  scales  to  a  discrete  five-value  risk  exposure. 


3.  Prototype  SE  Effectiveness  Risk  Tools 

As  a  means  to  test  the  utility  of  these  characteristics  for  assessing  systems  engineering  effectiveness, 
using  the  GQM  approach  outlined  above,  the  researchers  created  prototype  tools  that  might  be  used  to 
perform  periodic  evaluations  of  a  project,  similar  to  a  tool  used  in  conjunction  with  the  macro-risk 
model  described  above.  The  following  section  describes  this  prototype  implementation  in  further  detail. 


3.1  SE  Performance  Risk  Tool 

The  Systems  Engineering  Performance  Risk  Tool  (SEPRT)  is  an  Excel  spreadsheet-based  prototype 
focused  on  enabling  projects  to  determine  their  relative  risk  exposure  due  to  shortfalls  in  their  SE 
performance  relative  to  their  prioritized  project  needs.  It  complements  other  SE  performance 
effectiveness  assessment  capabilities  such  as  the  INCOSE  Leading  Indicators,  in  that  it  supports 
periodic  assessment  of  evidence  of  key  SE  function  performance,  as  compared  to  supporting  continuous 
assessment  of  key  project  SE  quantities  such  as  requirements  volatility,  change  and  problem  closure 
times,  risk  handling,  and  staffing  trends. 

The  operational  concept  of  the  SEPRT  tool  is  to  enable  project  management  (generally  the  Project 
Manager  or  his/her  designate)  to  prioritize  the  relative  impact  on  the  particular  project  of  shortfalls  in 
performing  the  SE  task  represented  in  each  question.  Correspondingly,  the  tool  enables  the  project 
systems  engineering  function  (generally  the  Chief  Engineer  or  Chief  Systems  Engineer  or  their 
designate)  to  evaluate  the  evidence  that  the  project  has  adequately  performed  that  task.  This 
combination  of  impact  and  risk  assessment  enables  the  tool  to  estimate  the  relative  project  risk  exposure 
for  each  question,  and  to  display  them  in  a  color-coded  Red- Yellow-Green  form. 

These  ideas  were  reviewed  in  workshops  with  industry,  government,  and  academic  participants 
conducted  in  March  and  May  2009,  with  respect  to  usability  factors  in  a  real  project  environment.  A 
consensus  emerged  that  the  scale  of  risk  impact  and  risk  probability  estimates  should  be  kept  simple  and 
easy  to  understand.  Thus  a  red,  yellow,  green,  and  grey  scale  was  suggested  to  code  the  risk  impact;  and 
a  corresponding  red,  yellow,  green,  and  blue  scale  to  code  the  risk  probability.  These  scales  are 
discussed  in  more  depth  below.  An  example  of  the  rating  scales,  questions,  and  calculated  risk  exposure 
in  the  prototype  tool  is  presented  below. 
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NOTE:  Impact  and  evidence/risk  ratings  should  be  done  independently.  The 
impact  rating  should  estimate  the  effect  a  failure  to  address  the  specified  item 
might  have  on  the  program.  The  evidence  rating  should  specify  the  qualtity  of 
evidence  that  has  been  provided,  which  demonstrates  that  the  specified  risk  Risk 
item  has  been  satisfactorily  addressed.  Exposure 


Goal  1: 


Concurrent  definition  of  system  requirements  and  solutions 


Understanding  of  stakeholder  needs:  capabilities,  operational  concept,  key 
performance  parameters,  enterprise  fit  (legacy) 

At  Milestone  A,  have  the  KPPs  been  identified  in  clear,  comprehensive,  concise  terms  that 
are  understandable  to  all  stakeholders? 

Has  a  CONOPS  been  developed  showing  that  the  system  can  be  operated  to  handle  both 
nominal  and  off-nominal  workloads,  to  meet  response  time  requirements,  and  generally 
to  meet  the  defined  KPPs? 

Has  the  ability  of  the  system  to  meet  mission  effectiveness  goals  been  verified  through 
the  use  of  modeling  and  simulation? 


Have  the  success-critical  stakeholders  been  identified,  their  roles  and  responsibilities 
negotiated,  and  their  needs  clearly  represented  by  the  KPPs  and  CONOPS? 

Have  issues  about  the  fit  of  the  system  into  the  stakeholders’  context  -  acquirers,  end 
users,  administrators,  interoperators,  maintainers,  etc.  --  been  adequately  explored? 


Figure  3.  Example  of  SEPRT  prototype 


Risk  impact  ratings  vary  from  a  critical  impact  of  shortfalls  in  performing  the  SE  task  in  question  (red) 
through  significant  impact  (yellow)  and  moderate  impact  (green)  to  little-no  impact  (gray),  as  illustrated 
in  Table  7.  These  relative  impact  ratings  enable  projects  to  tailor  the  evaluation  to  the  project’s  specific 
situation.  Thus,  for  example,  it  is  easy  to  “drop”  a  question  by  clicking  on  its  “No  Impact”  button,  but 
also  easy  to  restore  it  by  clicking  on  a  higher  impact  button.  The  rating  scale  for  the  impact  level  is 
based  on  the  user’s  chosen  combination  of  effects  on  the  project’s  likely  cost  overrun,  schedule  overrun, 
and  missing  percent  of  promised  over  actual  delivered  capability  (considering  there  are  various  tradeoffs 
among  these  quantities). 


Table  7.  Risk  impact  ratings 


Rating 

Cost/Schedule/Capability  Shortfall 

Little-No  impact  (Gray) 

0-2%  (1%  average) 

Moderate  impact  (Green) 

2-20%(ll%  average) 

Significant  impact  (Yellow) 

20-40%  (30%  average) 

Critical  impact  (Red) 

40-100%  (70%  average) 

Using  Question  1.1(a)  from  Figure  3  as  an  example,  if  the  project  were  a  back-room  application  for  base 
operations  with  no  mission-critical  key  performance  parameters  (KPPs),  its  impact  rating  would  be 
Little-No  impact  (Gray).  However,  if  the  project  were  a  C4ISR  system  with  several  mission-critical 
KPPs,  its  rating  would  be  Critical  impact  (Red). 
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The  Evidence/Risk  rating  is  the  project’s  degree  of  evidence  that  each  SE  effectiveness  question  is 
satisfactorily  addressed,  scored  (generally  by  the  project  Chief  Engineer  or  Chief  Systems  Engineer  or 
their  designate)  on  a  risk  probability  scale:  the  less  evidence,  the  higher  the  probability  of  shortfalls.  The 
evaluator  chooses  a  rating  based  on  the  probability  of  an  unsuccessful  outcome  in  performing  the  SE 
task  in  question,  as  noted  in  Table  8 


Table  8.  Risk  probability/evidence  ratings 


Likelihood  of  Shortfall 

Degree  of  evidence 

Probability  Range 

High  probability  (Red) 

Little-no  evidence 

P  =  0.4  - 1.0;  average  0.7 

Medium  probability  (Yellow) 

Weak  evidence 

P  =  0.2-  0.4;  average  0.3 

Low  probability  (Green) 

Partial  evidence 

P  =  0.02  -  0.2;  average  0.11 

Very  Low  probability  (Blue) 

Strong  and  externally 
validated  evidence 

P  =  0  -  0.02;  average  0.01 

Again,  using  Question  1.1(a)  from  Figure  3  as  an  example  analyzing  a  C4ISR  system  with  several 
mission-critical  KPPs,  then  a  lack  of  evidence  (from  analysis  of  current-system  shortfalls  and/or  the  use 
of  operational  scenarios  and  prototypes)  that  its  “KPPs  had  been  identified  at  Milestone  A  in  clear, 
comprehensive,  concise  terms  that  are  understandable  to  the  users  of  the  system”  would  result  in  a  High 
risk  probability,  while  strong  and  externally  validated  evidence  would  result  in  a  Very  Low  risk 
probability. 

Using  the  average  probability  and  impact  values  from  Table  7  and  Table  8,  the  relative  Risk  Exposure  = 
P(Risk)  *  Size(Risk)  implied  by  the  ratings  is  presented  in  Table  9. 


Table  9.  Average  risk  exposure  calculation 


Impact  //  Probability 

Very  Low 

Low 

Medium 

High 

Critical 

0.7 

7.7 

21 

49 

Significant 

0.3 

3.3 

9 

21 

Moderate 

0.11 

1.21 

3.3 

7.7 

Little-No  Impact 

0.01 

0.11 

0.3 

0.7 

The  prototype  tool  provides  a  customizable  mapping  of  each  impact/probability  pair  to  a  color-coded 
risk  exposure,  based  on  the  above  table.  For  each  question,  the  risk  exposure  level  is  determined  by  the 
combination  of  risk  impact  and  risk  probability,  and  a  corresponding  risk  exposure  color-coding  is 
selected,  which  ranges  from  red  for  the  highest  risk  exposure  to  green  for  the  lowest.  Figure  4  provides 
an  example  of  this  mapping  from  the  prototype  tool,  and  illustrates  the  resulting  risk  exposure  matrix  for 
the  selected  mapping. 
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Figure  4.  Impact/risk  mapping  to  risk  exposure 


The  current  tool  assigns  the  highest  (red)  risk  exposure  for  the  (Impact,  Probability)  combinations  of 
(Critical,  Medium),  (Critical,  High),  and  (Significant,  High).  It  assigns  a  medium-high  (orange)  risk 
exposure  for  the  (Impact,  Probability)  combinations  of  (Critical,  Low),  (Significant,  Medium),  and 
(Moderate,  Medium).  A  medium  (yellow)  risk  exposure  is  assigned  for  the  (Impact,  Probability) 
combinations  of  (Significant,  Low),  and  (Moderate,  Medium).  A  medium-low  (green)  risk  exposure 
results  from  the  (Impact,  Probability)  combinations  of  (Critical,  Very  Low),  (Moderate,  Low),  and 
(Little-No,  High).  Finally,  all  remaining  combinations  involving  Little-No  impact  or  Very  Low 
probability  are  assigned  the  lowest  risk  exposure  (green). 

As  seen  in  Figure  3,  the  risk  exposure  resulting  from  scoring  the  impact  and  risk  of  each  question  is 
presented  in  the  leftmost  column.  Based  on  suggestions  from  workshop  participants,  the  current  version 
of  the  tool  assigns  the  highest  risk  exposure  level  achieved  by  any  of  the  questions  in  a  CSF  as  the  risk 
exposure  for  the  overall  CSF.  This  maximum  risk  exposure  presented  in  the  rightmost  column  for  the 
CSF.  This  rating  method  has  the  advantages  of  being  simple  and  conservative,  but  might  raise  questions 
if,  for  example,  CSF  1.1  were  given  a  red  risk  exposure  level  for  one  red  and  four  greens,  and  a  yellow 
risk  exposure  level  for  five  yellows.  Experience  from  piloting  of  the  tool  has  suggested  refinements  to 
this  approach,  discussed  later  in  this  report. 


3.2  SE  Competency  Risk  Tool 

The  Systems  Engineering  Competency  Risk  Tool  (SECRT),  like  the  SEPRT  tool  described  above,  is  a 
prototype  expression  of  the  framework  for  evaluating  SE  competency.  Although  the  framework  is 
believed  to  be  complete,  the  question  set  supporting  the  framework  is  not  based  on  a  coverage  matrix 
analysis,  but  largely  based  on  the  SEPRT  framework  plus  the  additional  goal  and  five  CSFs  addressing 
Professional  and  Interpersonal  Skills.  SECRT’s  concise,  project-tailorable  content  complements  other 
more  detailed  SE  personnel  competency  frameworks  that  focus  more  on  SE  personnel  certification  or 
organizational  skills  coverage. 

The  general  design  of  the  SECRT  prototype  is  identical  to  that  of  SEPRT — an  Excel-based  spreadsheet. 
The  SECRT  impact  and  evidence  scales  are  the  same  as  SEPRT,  except  that  the  ratings  for  evidence 
score  the  relative  experience  and  competency  of  the  team  needed  to  fulfill  the  SE  function.  This  scale 
reflects  the  team’s  composite  experience  and  competency  level  with  respect  to  its  relevant  systems 
engineering  technical,  professional,  and  applications  domain  knowledge,  skills,  and  abilities.  Similar  to 
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SEPRT,  the  SECRT  evidence  ratings  range  from  a  High  probability  of  an  unsuccessful  outcome  in 
performing  the  SE  task  in  question  (red;  no  relevant  experience  and  competency),  through  Medium 
probability  (yellow;  some  relevant  experience  and  competency)  and  Low  probability  (green;  good 
relevant  experience  and  competency),  to  very  low  probability  of  an  unsuccessful  outcome  (blue;  expert 
relevant  experience  and  competency). 

The  following  sections  describe  and  use  the  SEPRT  tool  as  an  example,  though  the  discussion  applies 
equally  to  the  SECRT  tool.  It  is  expected  that  the  SECRT  framework  will  continue  to  evolve  and 
mature  in  future  research. 


4.  Description  and  Usage  of  Prototype  Tools 

The  prototype  Systems  Engineering  Performance  Risk  Tool  (SEPRT)  and  Systems  Engineering 
Competence  Risk  Tool  (SECRT)  tools  are  extendable  Excel  spreadsheets  organized  around  a  framework 
of  systems  engineering  goals,  contributing  critical  success  factor  questions,  and  detailed  metric 
questions.  Each  question  can  be  prioritized  for  relevance  to  the  particular  systems  engineering  effort, 
and  assessed  with  respect  to  the  degree  of  evidence  that  it  is  being  well  addressed. 

The  tools  are  intended  for  use  at  discrete  assessment  points  during  a  project’s  lifetime.  For  example,  the 
tools  might  be  used  to  review  SE  plans  or  preparations  for  major  milestone  reviews  to  assess  any 
shortfalls  in  SE  effectiveness.  The  SEPRT  tool  addresses  the  evidence  of  thoroughness  with  which  core 
systems  engineering  functions  are  being  performed.  The  SECRT  tool  assesses  the  evidence  of  whether 
sufficient  SE  team  personnel  competence  is  in  place  to  carry  out  the  functions.  Both  tools  treat  a 
shortfall  in  evidence  as  an  increased  risk  probability.  This  probability,  multiplied  by  the  relative  impact 
of  the  item  on  project  success,  produces  a  risk  exposure  quantity,  color-coded  for  identification  of  the 
risk  levels  of  SE  effectiveness  items. 

The  primary  objective  of  piloting  the  prototype  tools  is  to  determine  the  utility  of  the  evaluation 
frameworks  at  various  points  in  a  project's  life  cycle,  across  different  acquirers  and  application  domains. 
Data  of  interest  from  piloting  these  tools  includes  both  the  cost  in  effort  required  to  perform  the 
assessments,  and  the  value  obtained  from  performing  them.  It  is  not  an  objective  of  the  pilot  evaluations 
for  evaluators  to  externally  disclose  shortfalls  or  risks  in  the  projects  assessed,  although  the  researchers 
are  seeking  information  on  the  effects  of  using  the  tools. 


4.1  Tool  Overview 

The  SEPRT  and  SECRT  tools  are  Excel-based  prototypes.  The  tools  were  created  in  Excel  2007.  Users 
with  Excel  2003  will  have  the  same  functionality,  but  the  risk  exposure  color-coding  does  not  function. 
Macros  must  be  enabled  for  functionality.  The  SECRT  competency  evaluation  tool  operates  identically 
to  the  SEPRT  tool  described  below,  though  the  critical  success  factors  and  evaluation  questions  differ. 

Each  tool  identifies  high-level  Goals  that  must  be  met,  and  provides  four  or  five  Critical  Success  Factors 
that  support  each  goal.  Questions  then  explore  whether  the  critical  success  factors  are  being  met.  Each 
question  is  evaluated  against  two  separate  scales:  evidence  and  impact.  Less  evidence  is  equated  to 
higher  risk.  An  impact  rating  for  each  question  allows  the  evaluator  to  adjust  the  weighting  of  the 
question  for  that  particular  project. 
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It  is  recommended  that  the  impact  rating  and  evidence  scores  be  determined  by  independent  reviewers. 
For  instance,  the  project  or  program  manager  or  their  designate  might  provide  the  impact  ratings,  and 
project  chief  engineer  or  chief  systems  engineer  or  their  designate  might  provide  the  evidence  ratings. 

Figure  5  illustrates  the  rating  scale  for  impact  and  evidence  on  each  question.  In  the  leftmost  set  of 
selections,  the  evaluator  selects  an  appropriate  weighting  for  the  impact,  ranging  from  Critical  impact 
(red)  to  Little-No  impact  (gray).  Similarly,  the  rightmost  selection  set  indicates  the  degree  of  evidence 
available  that  supports  the  evaluation  of  each  question,  where  red  implies  little-or-no  evidence  has  been 
found  to  support  the  conjecture,  and  blue  implies  that  independent  experts  have  validated  the  evidence. 
Users  make  selections  by  clicking  on  the  appropriately  colored  boxes  for  each  question. 
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Figure  5.  Detail  of  impact  and  evidence  rating 


As  seen  in  Figure  6,  the  impact  and  evidence  scores  for  each  critical  success  factor  are  rolled  up  into  an 
overall  risk  exposure,  which  again  is  represented  as  a  simple  red-orange-yellow-light  green-green 
indicator  (for  Excel  2003  users,  risk  exposure  is  5,  4,  3,  2,  1,  respectively).  The  overall  risk  exposure  is 
the  maximum  of  the  risk  exposures  determined  from  the  responses  to  the  individual  questions  that 
support  each  critical  success  factor.  The  “rationale”  column  may  be  used  to  record  the  source  of 
evidence  for  later  review.  The  “reset”  button  clears  the  impact  and  evidence  ratings  for  the  entire 
document. 
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Figure  6.  Detail  of  risk  exposure  rollup  by  CSF 


Risk  exposure  (RE)  is  calculated  by  multiplying  the  risk  impact  by  the  probability  of  risk  (exposure  = 
impact  *  p(risk)).  Since  the  impact  and  probability  of  risk  are  represented  here  as  discrete  quantities, 
however,  a  different  approach  was  used  to  determine  the  risk  exposure.  Figure  7  is  an  excerpt  from  the 
“RE  Map”  tab  of  the  SEPRT  and  SECRT  tool  spreadsheets.  On  this  tab,  each  combination  of  impact 
and  p(risk) — where  zero  represents  little-no  impact/little-no  risk,  and  three  represents  critical 
impact/high  risk,  as  shown  by  the  “Scale” — may  be  assigned  a  value  from  one  (green)  to  five  (red)  by 
filling  in  the  “Color”  column.  The  risk  exposure  matrix  resulting  from  these  choices  is  automatically 
shown  on  the  right,  in  a  format  similar  to  the  five-by-five  representation  commonly  used  in  risk  analysis. 
In  this  example,  the  combination  of  “no  impact”  (impact=0)  and  “independently  validated  evidence” 
(p(Risk)=3)  result  in  a  medium-low  (light-green=2)  risk  exposure.  The  values  in  the  “Color”  column 
may  be  altered  to  suit  the  needs  of  the  program  being  evaluated. 


Figure  7.  Detail  of  risk  exposure  mapping 


4.2  Tool  Tailoring  and  Extension 

Being  developed  in  Excel,  the  prototype  tools  are  relatively  simple  to  tailor  and  extend,  to  explore 
additional  framework  concepts.  Questions  can  be  added  to  a  CSF  by  copying  and  pasting  an  existing 
question  and  modifying  it.  The  formulas  for  computing  risk  exposure  are  straightforward,  and  can  be 
similarly  copied  and  adjusted  to  refer  to  the  new  questions. 

The  risk  exposure  mapping  itself  is  designed  to  be  configurable  by  the  individual  project.  Using  the 
“RE  Mapping”  tab,  as  described  in  Figure  7  above,  combinations  of  risk  impact  and  probability  can  be 
mapped  to  different  risk  exposure  ratings,  simply  by  modifying  the  RE  numbers  in  the  “Color”  column. 

Based  on  feedback  from  the  pilot  evaluations,  it  may  be  desirable  in  the  future  to  modify  the  EM 
frameworks  to  be  less  specific  to  DoD  terminology  and  milestones.  Although  the  exact  phrasing  of 
goals,  CSFs,  and  questions  is  a  topic  for  further  research,  the  tools  are  easily  modifiable  to  reflect  the 
desired  changes. 
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There  is  some  interest  in  adapting  the  frameworks  to  address  the  specific  needs  of  particular  application 
domains.  For  example,  the  SE  needs  for  engineering  secure  systems  are  more  specific  than  the  general 
SE  needs  expressed  in  the  present  framework.  Similarly,  organizations  focused  on  ground,  sea,  air,  and 
space  missions  would  benefit  from  domain-specific  extensions.  Future  research  intends  to  explore  how 
the  framework  might  be  adapted  to  such  specific  needs,  and  to  adapt  the  prototype  tools  to  enable 
piloting  of  these  future  efforts. 


5.  Summary  of  Framework  and  Tool  Evaluations 

The  researchers  solicited  pilot  evaluations  of  the  EM  performance  and  competency  frameworks,  using 
the  prototype  SEPRT  and  SECRT  tools,  from  industry,  government  agencies,  and  academic  participants. 
Because  the  task  re-scoping  permitted  only  a  single  round  of  piloting,  these  initial  evaluations  were 
conducted  against  historical  projects  and  case  studies.  In  addition,  the  University  of  Maryland  (UMD) 
Fraunhofer  Center  (FC)  began  preliminary  evaluations  against  the  Systemic  Analysis  Database  (SADB), 
compiled  by  OUSD  (AT&L).  This  evaluation  approach  allowed  analysis  of  the  effectiveness  of  the 
frameworks  with  respect  to  historical  success  and  failures  of  the  subject  projects. 

The  tools  were  successfully  piloted  against  five  DoD  projects,  one  NASA  project,  and  one  commercial 
project.  They  were  also  analyzed  by  two  industrially-experienced  colleagues  against  detailed  case 
studies  of  a  number  of  DoD  and  commercial  projects.  The  application  domains  piloted  included  space, 
medical  systems,  logistics,  and  systems-of-systems.  Results  of  the  pilot  evaluations  were  reported 
through  a  web-based  survey  tool  and  detailed  follow-up  interviews,  while  the  case  study  evaluations 
were  reported  through  detailed  comments  from  the  reviewers. 

Evaluations  were  generally  positive,  and  the  frameworks  were  found  to  be  useful  across  all  project 
phases  except  Production,  and  against  all  systems  types  except  “legacy  development.”  The  consensus  of 
reviewers  was  that  the  frameworks  would  be  most  useful  in  the  System  Development  &  Demonstration 
(SDD)  phase,  and  generally  more  useful  in  early  phases  than  later.  It  was  noted,  however,  that  in 
systems  developed  using  evolutionary  strategies,  such  “early”  phases  recur  throughout  the  development 
cycle,  extending  the  usefulness  of  the  frameworks.  The  evaluations  were  reported  to  take  2-5  hours  to 
complete  for  persons  familiar  with  the  projects,  with  materials  that  were  readily  at  hand. 

Some  concerns  reported,  particularly  by  the  NASA  reviewers,  were  that  the  frameworks  were  too  DoD- 
centric  in  their  terminology,  milestones,  and  acquisition  frameworks.  These  reviewers  were  sufficiently 
familiar  with  DoD  processes  to  allow  analysis  of  the  NASA  projects,  but  suggested  that  the  CSFs  and 
questions  be  modified  to  be  more  general.  In  reviewing  case  study  material,  some  evaluators  reported 
that  the  EM  framework  was  not  specific  to  any  particular  problem  domain.  Finally,  several  evaluators 
reported  that  the  frameworks  generated  too  many  high-risk  findings,  which  might  make  the  results  too 
overwhelming  to  take  action. 

These  concerns  were  partially  mitigated  with  suggestions  from  the  evaluators.  Although  the  general 
nature  of  the  EM  frameworks  was  intentional  in  this  portion  of  the  research,  it  suggests  that  tailoring 
might  be  useful  to  uncover  domain-specific  shortcomings  in  SE  functions.  Two  approaches  might  be 
used  with  respect  to  the  DoD-specific  nature  of  the  frameworks:  the  frameworks  might  be  edited  to  use 
more  generic  terms,  or  new  versions  tailored  to  other  agencies  and  domains.  With  respect  to  the 
frameworks  uncovering  too  many  high-risk  findings,  the  impact  scales  have  been  adjusted  to  make  the 
adjectives  better  correspond  to  the  quantitative  impacts  (Critical — Significant — Moderate — Little-or-No 
vs.  High-Medium-Low-No  impact),  and  a  longer  risk  exposure  scale  developed  to  allow  more  nuanced 
results. 
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Details  regarding  use  of  the  SEPRT  (performance)  tool  are  as  follows.  It  was  found  to  be  particularly 
useful  in  SDD,  somewhat  useful  in  early  phases,  and  less  useful  in  later  phases.  This  result  is  somewhat 
expected,  given  the  “early-phase”  emphasis  of  the  materials  used  as  sources  for  the  frameworks,  and 
suggests  that  further  research  might  examine  the  SE  strategies  that  are  important  in  later  life-cycle 
phases,  such  as  testing  and  configuration  management.  The  evaluators  also  report  that  the  SEPRT 
effectiveness  ranges  from  very  effective  to  somewhat  effective,  with  the  majority  reporting  it  as 
effective.  Only  analysis  of  one  DoD  legacy  system  project  reported  the  tool  as  ineffective,  which 
suggests  that  the  issues  facing  such  projects  may  need  to  be  examined  more  closely.  Even  with  respect 
to  DoD  systems,  some  evaluators  reported  issues  with  the  terminology  used,  which  might  be  cleared  up 
with  careful  editing. 

With  respect  to  the  SECRT  (competency),  the  evaluators  also  found  the  tool  most  useful  in  earlier  life- 
cycle  phases,  rather  than  later.  The  usefulness  was  rated  between  effective  and  somewhat  effective. 
Although  the  SECRT  is  less  well  developed  than  the  SEPRT,  one  reviewer  noted  that  the  shortfalls 
noted  by  the  SECRT  might  well  be  used  to  help  justify  and  explain  the  need  for  stronger  SE  capabilities 
to  program  management  and  acquirers.  Another  reviewer  observed  that  the  choice  of  person  performing 
the  competency  evaluation  was  critical,  as  it  is  difficult  for  a  non-technical  evaluator  to  judge 
competency.  Finally,  one  comment  that  is  more  a  judgment  of  state  of  practice  is  that,  “to  be  effective, 
program  management  must  have  some  control  over  who  is  assigned.” 

Several  reviewers  comment  that  the  simple  red,  yellow,  green,  and  blue/gray  choices  for  impact  and 
evidence  ratings  might  be  too  granular.  However,  the  initial  rating  scales  used  a  1-5  range,  which 
workshop  participants  judged  as  too  complex.  The  resulting  granularity  of  risk  exposure  (RE)  results 
might  be  mitigated  using  several  techniques.  The  RE  calculation  has  been  broadened  to  allow  medium- 
high  (orange)  and  medium-low  (light  green)  results,  which  reduces  the  clumping  of  all  results  into 
medium  (yellow)  and  high  (red).  The  impact  scale  has  also  been  clarified  to  make  the  lowest  choice 
“little  or  no”  impact,  rather  than  “no”  impact.  Since  the  prototype  tools  presently  use  the  maximum  RE 
of  all  questions  in  a  CSF,  another  suggestion  was  to  provide  a  count  of  the  number  of  REs  at  this 
maximum  level,  as  a  “red  (4)”  is  clearly  worse  than  a  “red  (1).”  All  but  the  last  of  these  suggestions 
have  already  been  retrofitted  into  the  SEPRT  and  SECRT  tools,  although  the  changes  have  not  yet  been 
re-piloted. 

The  evaluations  indicate  that  the  concept  of  “evidence-based”  information  is  not  well  understood,  and 
may  require  further  explanation  and  example.  For  example,  one  reviewer  asks  specifically  what 
constitutes  “sufficient”  evidence.  One  answer  is  evidence  that  an  independent,  expert  reviewer  would 
find  sufficient  as  a  compelling  and  objectively  verifiable  argument  of  feasibility.  This  is  a  slight 
modification  of  the  original  definition,  where  the  word  “external”  was  used  rather  than  “independent,” 
as  one  reviewer  observed  that  for  some  domains,  external  reviewers  would  not  have  sufficient  context  to 
perform  knowledgeable  assessments.  It  might  also  be  necessary  to  supplement  the  framework  question 
set  with  material  that  describes  the  intended  goal  for  each  question,  so  that  evidence  can  be  judged 
against  this  goal. 

In  summary,  the  framework  and  prototype  tools  have  been  shown  to  be  largely  efficacious  for  pilot 
projects  done  by  familiar  experts  in  a  relatively  short  time,  in  identifying  characteristics  of  SE  efforts 
that,  inadequately  performed,  will  likely  lead  to  difficulties  with  the  program.  It  remains  to  demonstrate 
how  well  the  framework  and  tools  will  perform  on  in-process  MDAPs  with  multiple  missions, 
performers,  and  independent  expert  assessors. 
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C.  DoD  MDAP  SE  EM  Concepts  of  Operation 


The  SEPRT  and  SECRT  framework  and  tools  provide  a  way  for  MDAPs  (and  other  projects)  to  identify 
the  major  sources  of  program  risk  due  to  SE  shortfalls.  This  section  provides  concepts  of  operation  for 
applying  the  tools  at  major  milestones,  and  at  other  points  where  other  SE  EMs  such  as  the  fNCOSE 
Leading  Indicators  have  identified  likely  problem  situations  and  need  further  understanding  of  the 
problem  sources  and  their  relative  degrees  of  risk. 

Section  C.l  establishes  the  primary  criteria  for  satisfactory  program  evidence,  and  provides  a  vehicle  for 
capturing  the  evidence  called  a  Feasibility  Evidence  Description.  Section  C.2  provides  representative 
MDAP  operational  scenarios  that  show  how  the  determination  of  SEPRT  and  SECRT  impact  priorities 
can  be  done  collaboratively  by  a  program’s  sponsors  and  performers  at  various  life  cycle  points, 
enabling  a  cooperative  rather  than  an  adversarial  approach  to  system  definition  and  development. 
Section  C.3  shows  how  the  use  of  feasibility  evidence  as  a  first-class  deliverable  enables  programs  to 
plan  and  control  progress  toward  successful  passage  of  SEPRT-  and  SECRT-based  reviews.  Section 
C.4  describes  a  process  for  conducting  such  reviews,  and  summarizes  successful  experiences  in 
applying  such  processes. 


1.  Evidence  Criteria  and  Review  Milestone  Usage 

Having  shown  in  Section  A.3  that  the  regions  of  high  payoff  for  evidence-based  specifications,  plans, 
assessments,  and  reviews  are  extensive  and  enterprise-critical,  it  is  now  important  to  define  the  criteria 
for  the  evidence  that  will  be  associated  with  the  system  development  specifications  and  plans,  and 
reviewed  via  the  SEPRT  and  SECRT.  The  criteria  are  extensions  to  those  for  the  Anchor  Point  (AP) 
milestones  defined  in  [7]  and  adopted  by  the  Rational  Unified  Process  (RUP)  [18], [25],  The  extensions 
have  been  incorporated  into  the  recently  developed  Incremental  Commitment  Model  (ICM);  details  of 
this  model  can  be  found  in  [10], [23].  Also,  comparable  benefits  can  be  obtained  by  adding  such  criteria, 
processes,  and  incentive  structures  to  traditional  acquisition  methods  and  reviews. 

The  evidence  criteria  are  embodied  in  a  Feasibility  Evidence  Description  (FED).  It  includes  evidence 
provided  by  the  developer  and  validated  by  independent  experts  that,  if  the  system  is  built  to  the 
specified  architecture  it  will: 

1 .  Satisfy  the  specified  operational  concept  and  requirements,  including  capability,  interfaces,  level  of 
service,  and  evolution 

2.  Be  buildable  within  the  budgets  and  schedules  in  the  plan 

3.  Generate  a  viable  return  on  investment 

4.  Generate  satisfactory  outcomes  for  all  of  the  success-critical  stakeholders 

5.  Identify  shortfalls  in  evidence  as  risks,  and  cover  them  with  risk  mitigation  plans 

An  FED  does  not  assess  a  single  sequentially  developed  system  definition  element,  but  the  consistency, 
compatibility,  and  feasibility  of  several  concurrently  engineered  elements.  To  make  this  concurrency 
work,  a  set  of  Anchor  Point  milestone  reviews  are  performed  to  ensure  that  the  many  concurrent 
activities  are  synchronized,  stabilized,  and  risk-assessed  at  the  end  of  each  phase.  Each  of  these  reviews 
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is  focused  on  developer-produced  and  expert-validated  evidence,  documented  in  the  FED  (or  by  pointers 
to  the  results  of  feasibility  analyses),  to  help  the  system’s  success-critical  stakeholders  determine 
whether  to  proceed  into  the  next  level  of  commitment.  Hence,  they  are  called  Commitment  Reviews. 

The  FED  is  based  on  evidence  from  simulations,  models,  or  experiments  with  planned  technologies  and 
increasingly  detailed  analysis  of  development  approaches  and  project  productivity  rates.  The 
parameters  used  in  the  analyses  should  be  based  on  measured  component  performance  or  on  historical 
data  showing  relevant  past  performance,  cost  estimation  accuracy,  and  actual  developer  productivity 
rates. 

As  determined  by  independent  experts  using  the  SEPRT  and  SECRT  questions,  a  shortfall  in  feasibility 
evidence  indicates  a  level  of  program  execution  uncertainty  and  a  source  of  program  risk.  It  is  often  not 
possible  to  fully  resolve  all  risks  at  a  given  point  in  the  development  cycle,  but  known,  unresolved  risks 
need  to  be  identified  and  covered  by  risk  management  plans,  including  the  necessary  staffing  and 
funding  to  address  them.  The  nature  of  the  evidence  shortfalls,  the  strength  and  affordability  of  the  risk 
management  plans,  and  the  stakeholders’  degrees  of  risk  acceptance  or  avoidance  will  determine  their 
willingness  to  commit  the  necessary  resources  to  proceed.  A  program  with  risks  is  not  necessarily  bad, 
particularly  if  it  has  strong  risk  management  plans.  A  program  with  no  risks  may  be  high  on 
achievability,  but  low  on  ability  to  produce  a  timely  payoff  or  a  competitive  advantage. 

An  FED  needs  to  be  more  than  just  traceability  matrices  and  PowerPoint  charts.  Evidence  can  include 
results  of: 

•  Prototypes:  of  networks,  robots,  user  interfaces,  COTS  interoperability 

•  Benchmarks:  for  performance,  scalability,  accuracy 

•  Exercises:  for  mission  performance,  interoperability,  security 

•  Models:  for  cost,  schedule,  performance,  reliability;  tradeoffs 

•  Simulations:  for  mission  scalability,  performance,  reliability 

•  Early  working  versions:  of  infrastructure,  data  fusion,  legacy  compatibility 

•  Previous  experience 

•  Combinations  of  the  above 

Not  only  does  the  evidence  need  to  be  produced,  but  it  needs  to  be  validated  by  independent  experts. 
These  experts  need  to  determine  the  realism  of  assumptions,  the  representativeness  of  scenarios,  the 
thoroughness  of  analysis,  and  the  coverage  of  key  off-nominal  conditions.  At  least  one  DoD  MDAP  has 
explicitly  included  a  FED  Data  Item  Description  in  its  list  of  deliverables. 


2.  Use  of  the  SERC  SE  EM  Capabilities  in  Evidence-Based  SE 

This  section  shows  how  the  EMs  can  be  used  to  reach  sponsor-performer  consensus  on  the  relative 
impact  of  each  EM  upon  the  project  outcome,  and  of  the  resources  required  to  achieve  it.  These  then 
serve  as  a  consensus-based  set  of  criteria  that  will  be  used  to  measure  evidence  of  the  project’s  SE 
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effectiveness,  which  is  then  reinforced  by  becoming  a  significant  determinant  of  the  performer’s  award 
fee.  Shortfalls  in  evidence  are  uncertainties  or  probabilities  of  loss,  which  when  multiplied  by  the 
relative  impact  or  size  of  loss,  become  measures  of  project  risk.  These  then  require  risk  management 
plans,  employing  the  major  risk  mitigation  options  of  buying  information,  risk  avoidance,  risk  transfer, 
risk  reduction,  or  risk  acceptance.  Three  early-SE  scenarios  are  provided: 

1.  At  Milestone  A:  Milestone  Decision  Authority  (MDA)  Review  of  Acquirer  Plans 

2.  Contract  Negotiation:  MDAP  Acquirer  and  Developer 

3.  Project  Execution:  MDAP  Developer  Manager  and  Performers. 

2.1  Scenario  1:  MDA  Review  of  Acquirer  Plans  at  Milestone  A 

Step  1 .  The  Acquirer  submits  a  proposed  acquisition  plan  to  the  MDA,  along  with  the  SEPRT 
and  SECRT  evidence  ratings  and  risk  mitigation  approaches. 

Step  2.  The  MDA  has  independent  experts  review  the  SEPRT  and  SECRT  ratings.  A  major 
finding  is  that  no  Analysis  of  Alternatives  has  been  performed.  Only  one  alternative  has  been 
analyzed,  but  the  relative  risk  shown  in  SEPRT  is  low  because  the  Acquirer  has  assigned  it  a 
Little  or  No  Impact  rating. 

Step  3,  Case  1.  The  Acquirer  agrees  that  the  capability  needed  is  critical,  but  that  it  is  needed 
quickly  for  a  relatively  unique  and  short-term  threat,  and  that  evidence  is  available  that  a 
solution  involving  Alternative  A  will  be  sufficient.  The  MDA  concurs,  and  gives  approval  to 
rapidly  proceed  with  Alternative  A. 

Step  3,  Case  2.  The  Acquirer  states  that  a  Defense  Advanced  Research  Projects  Agency 
(DARPA)  demo  of  the  Alternative  X  technology  has  shown  proof  of  principle  of  its  feasibility, 
and  that  all  that  is  needed  is  to  implement  Alternative  X  for  the  general  case.  The  independent 
experts  conclude  that  the  proof  of  principle  demo  provided  no  evidence  of  the  solution's 
scalability  or  ability  to  work  in  degraded  battle  conditions.  The  MDA  does  not  give  an  approval 
to  proceed  with  Alternative  X,  but  directs  the  Acquirer  to  resubmit  using  a  Competitive 
Prototyping  acquisition  approach. 

In  each  case,  the  MDA  and  Acquirer  agree  on  an  acceptable  approach.  In  Case  1,  the  outcome  is  a 
timely  and  acceptable  solution.  In  Case  2,  Competitive  Prototyping  is  used  as  a  way  of  buying 
information  to  reduce  risk.  At  the  end  of  the  first  round  of  prototyping,  no  competitor  may  be  able  to 
develop  a  scalable  and  robust  solution,  and  the  acquisition  can  be  deferred  until  the  technology  is  more 
mature.  Or  it  may  be  that  one  or  more  competitors  have  produced  scalable  and  robust  solutions,  and 
another  round  of  prototyping  will  determine  the  best  approach  and  supplier. 


2.2  Scenario  2:  Acquirer  and  Winning-Prototype  Developer  Contract  Negotiation 

Step  1.  The  Acquirer  specifies  SEPRT  Critical  Success  Factor  1.2(d),  “Have  the  claimed  quality 
of  service  guarantees  been  validated?”  to  have  Critical  impact,  and  thus  to  be  a  major 
determinant  of  the  Performer’s  award  fee  at  each  review  milestone. 
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Step  2.  The  Developer  is  the  winning  competitive  prototyping  developer,  and  clearly  the 
performer  of  choice.  Their  response  is,  “We  agree  that  quality  of  service  evidence  is  critical,  as 
is  addressing  it  before  committing  to  functional  requirements.  But  we  have  tied  our  plans  and 
budgets  to  the  first  milestone  in  your  contract,  a  System  Functional  Requirements  Review  that 
ties  our  progress  payments  and  award  fees  to  just  specifying  functionality.  If  you  agree  that 
early  QoS  evidence  is  critical,  we  need  to  find  a  way  to  emphasize  this  in  the  contract.” 

Step  3.  The  Acquirer  responds,  “Thanks.  That  legacy  contract  clearly  undercuts  our  intent  to  do 
evidence-based  concurrent  engineering,  and  sets  us  up  for  late  overruns.  We’ll  redo  it  and  your 
SE  plans  and  budgets.  Next  time,  we’ll  address  contracting  compatibility  earlier.” 

2.3  Scenario  3:  Acquirer  and  Developer  Contract  Performance 

Having  agreed  on  a  revised  contract  and  increased  early-SE  budget  and  schedule,  justified  by  the 
rework-avoidance  business  case  in  Section  A.3,  the  Acquirer  and  Developer  use  Figure  8  as  a  basis  for 
proceeding.  Since  the  system  is  not  a  quick-response  development  for  a  relatively  unique  and  short¬ 
term  threat,  the  opportunistic  development  branch  is  not  chosen.  The  evaluation  of  SE  plans  and 
staffing  results  in  the  independent  evaluators  indicating  that  the  plans  and  staffing  are  sound,  but  also 
identifying  an  available  mission  simulator  that  can  increase  the  cost-effectiveness  of  the  evidence 
generation,  which  the  Developer  incorporates  into  the  plans. 

During  development,  the  project  uses  the  INCOSE  Leading  Indicators  capability  to  flag  any  progress 
indicators  that  exceed  a  set  of  control  limits  agreed  upon  by  the  Acquirer  and  Developer.  At  some  point, 
a  Leading  Indicator  shows  that  progress  is  behind  schedule  in  modifying  the  mission  simulator.  A 
specialized  SEPRT  assessment  of  the  simulator  subproject  finds  that  difficulties  have  been  encountered 
in  getting  the  simulator  to  provide  evidence  that  the  system  can  handle  some  off-nominal  threat 
scenarios  that  the  simulator  was  not  designed  to  handle.  A  quick  risk  mitigation  plan  is  developed  to 
bring  aboard  some  experts  to  rework  the  simulator  in  time  to  be  used  for  the  off-nominal  scenarios. 
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Figure  8.  Project  SysE  EM  Operational  Concept  (for  each  stage  of  system  definition  and 

development) 


3.  FED  Development  Process  Framework 

The  most  important  characteristic  of  evidence-based  system  specifications  and  plans  is  that: 

•  If  the  evidence  does  not  accompany  the  specifications  and  plans,  the  specifications  and  plans  are 
incomplete. 

Thus,  event-based  reviews,  where  the  event  is  defined  as  production  of  the  specifications  and  plans, 
need  to  be  replaced  by  evidence-based  reviews. 

This  does  not  mean  that  the  project  needs  to  spend  large  amounts  of  effort  in  documenting  evidence  of 
the  feasibility  of  a  simple  system.  The  appropriate  level  of  detail  for  the  contents  of  the  FED  is  based  on 
the  perceived  risks  and  criticality  of  the  system  to  be  developed.  It  is  NOT  a  “one  size  fits  all”  process, 
but  rather  a  framework  to  help  developers  and  stakeholders  determine  the  appropriate  level  of  analysis 
and  evaluation.  As  with  reused  specifications  and  plans,  evidence  can  be  appropriately  reused.  If  a 
more  complex  system  than  the  one  being  reviewed  has  been  successfully  developed  by  the  same  team,  a 
pointer  to  the  previous  project’s  evidence  and  results  will  be  sufficient. 

Table  10  outlines  a  process  that  can  be  used  for  developing  feasibility  evidence.  The  process  clearly 
depends  on  having  the  appropriate  work  products  for  the  phase  (Step  A).  As  part  of  the  engineering 
work,  high-priority  feasibility  assurance  issues  are  identified  that  are  critical  to  the  success  of  the  system 
development  program  (Step  B).  These  are  the  issues  for  which  options  are  explored,  and  potentially 
viable  options  further  investigated  (Step  C).  Clearly,  these  and  the  later  steps  are  not  performed 
sequentially,  but  concurrently. 
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Table  10.  Steps  for  Developing  a  FED 


Step 

Description 

Examples/Detail 

A 

Develop  phase  work-products/artifacts 

For  a  Development  Commitment  Review  (DCR), 
this  would  include  the  system’s  operational 
concept,  prototypes,  requirements,  architecture,  life 
cycle  plans,  and  associated  assumptions 

B 

Determine  most  critical  feasibility 
assurance  issues 

Issues  for  which  lack  of  feasibility  evidence  is 
program-critical 

C 

Evaluate  feasibility  assessment  options 

Cost-effectiveness;  necessary  tool,  data,  scenario 
availability 

D 

Select  options,  develop  feasibility 
assessment  plans 

The  list  of  options  at  the  end  of  Section  C.l 
(prototypes,  benchmarks,  exercises,  etc)  is  a  good 
starting  point 

E 

Prepare  FED  assessment  plans  and 
earned  value  milestones 

The  plans  include  the  enablers  in  Step  G 

F 

Begin  monitoring  progress  with  respect 
to  plans 

Also  monitor  changes  to  the  project,  technology, 
and  objectives,  and  adapt  plans 

G 

Prepare  evidence-generation  enablers 

Assessment  criteria 

Parametric  models,  parameter  values,  bases  of 
estimate 

COTS  assessment  criteria  and  plans 

Benchmarking  candidates,  test  cases 
Prototypes/simulations,  evaluation  plans,  subjects, 
and  scenarios 

Instrumentation,  data  analysis  capabilities 

H 

Perform  pilot  assessments;  evaluate  and 
iterate  plans  and  enablers 

Short  bottom-line  summaries  and  pointers  to 
evidence  files  are  generally  sufficient 

I 

Assess  readiness  for  Commitment 
Review 

Shortfalls  identified  as  risks  and  covered  by  risk 
mitigation  plans 

Proceed  to  Commitment  Review  if  ready 

J 

Hold  Commitment  Review  when  ready; 
adjust  plans  based  on  review  outcomes 

See  Commitment  Review  process  overview  below. 

NOTE:  “Steps”  are  denoted  by  letters  rather  than  numbers  to  indicate  that  many  are  done 
concurrently. 

Since  the  preliminary  design  and  plans  are  incomplete  without  the  FED,  it  becomes  a  first-class  project 
deliverable.  This  implies  that  it  needs  a  plan  for  its  development,  and  that  each  task  in  the  plan  needs  to 
be  assigned  an  appropriate  earned  value.  If  possible,  the  earned  value  should  be  based  on  the  potential 
risk  exposure  costs,  not  the  perceived  available  budget. 

Besides  monitoring  progress  on  developing  the  system,  the  project  needs  to  monitor  progress  on 
developing  the  feasibility  evidence.  This  implies  applying  corrective  action  if  progress  falls  behind  the 
plans,  and  adapting  the  feasibility  evidence  development  plans  to  changes  in  the  project  objectives  and 
plans.  If  evidence  generation  is  going  to  be  complex,  it  is  generally  a  good  idea  to  perform  pilot 
assessments.  The  preparations  for  the  commitment  review  are  discussed  next. 
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1.  Commitment  Review  Process  Overview 


Figure  9  highlights  the  activities  that  need  to  be  performed  in  preparation  for  the  review,  the  actual 
review,  as  well  as  the  post-review  activities  and  follow-up.  The  entry  criteria  include  ensuring  that  the 
feasibility  evidence  preparation  has  been  successfully  tracking  its  earned  value  milestones.  The  inputs 
include  preparing  the  domain  extensions  to  the  core  SEPRT  and  SECRT  framework  and  tools, 
identifying  committed  expert  reviewers  for  each  of  the  review  questions,  and  familiarizing  them  with  the 
SEPRT-SECRT  review  process. 

The  review  meeting  will  include  not  only  the  developer  SEs  and  the  expert  reviewers,  but  also  the 
stakeholder  upper-management  decision-makers,  who  will  need  some  context-setting  before  the 
developer  responses  to  reviewer  issues  are  discussed.  The  review  exit  criteria  and  tasks  include  key 
stakeholder  concurrence  on  the  way  forward  and  commitment  to  support  the  next  phase,  as  well  as 
action  plans  and  risk  mitigation  plans  for  the  issues  identified. 


Figure  9.  Overview  of  Commitment  Review  Process 


4.1  Examples  of  Successful  Experiences  with  Evidence-Based  Reviews 

AT&T  and  its  spinoffs  (Telcordia,  Lucent,  Avaya,  and  regional  Bell  companies)  have  been  successfully 
using  versions  of  evidence-based  reviews  since  1988.  On  average,  there  has  been  a  10%  savings  per 
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reviewed  project,  with  substantially  larger  savings  on  a  few  reviewed  projects.  More  detail  on  their 
Architecture  Review  experience  is  in  [19]. 

The  million-line  TRW  CCPDS-R  project  summarized  in  [25]  by  Walker  Royce,  and  several  similar 
TRW  follow-on  projects  were  further  successful  examples.  Evidence-based  anchor  point  milestone 
reviews  are  also  an  integral  part  of  the  Rational  Software  Process,  with  many  successful 
implementations  [26],  although  a  good  many  RUP  applications  have  been  unable  to  succeed  because  of 
the  type  of  unaddressed  contractual  constraints  discussed  in  Section  2.2. 

The  highly  successful  Abbott  Laboratories’  next  generation  intravenous  infusion  pump  documented  in 
chapter  5  of  [23]  is  a  good  commercial  example  of  evidence-based  specifications  and  plans. 


D.  Pilot  SEPRT  and  SECRT  Evaluation  Processes  and  Lessons  Learned 

1.  Introduction 

SEPRT  and  SECRT  were  reviewed  by  NASA’s  Marshall  Space  Flight  Center  (MSFC)  to  provide  an 
external  assessment  of  the  potential  application  of  the  tools  in  a  non-DoD  development  environment  and 
to  provide  an  external  review  of  the  tools  by  product  developers  working  in  a  similar  complex  system 
environment.  The  review  team  was  comprised  a  senior  systems  engineer  and  senior  project  manager  at 
MSFC,  supported  by  two  researchers  from  The  University  of  Alabama  in  Huntsville  (UAH).  Data  from 
this  review  was  used  as  inputs  to  the  sponsor  workshops  and  in  the  development  of  proposals  for 
extensions  to  this  research  effort. 

2.  Methodology 

The  review  of  the  SEPRT  and  SECRT  tools  was  done  in  four  phases.  Phase  one  was  a  briefing  to  the 
assessment  team  on  the  research  initiatives,  a  review  of  the  SEPRT  and  SECRT  tools,  and  a  discussion 
on  how  the  review  data  would  be  collected.  Phase  two  was  an  initial  assessment  of  the  tools  by  a  senior 
systems  engineer.  Phase  three  was  a  detailed  assessment  of  the  tools  by  the  entire  team.  Phase  four  was 
the  reporting  of  the  assessment  results  by  interview  and  by  completing  a  web-based  survey. 

The  phase  one  briefing  was  a  ninety  minute  session  that  included  both  the  MSFC  and  UAH  team 
members,  ft  was  held  at  MSFC.  The  briefing  included  an  overview  of  the  SERC  and  a  description  of 
the  specific  research  task  that  the  assessment  was  part  of.  A  review  of  the  data  collection  methods  was 
included.  The  MSFC  team  noted  that  they  would  prefer  to  discuss  their  review  findings  before 
completing  the  web-based  survey,  so  the  research  plan  was  modified  to  include  an  interview  in  phase 
four.  The  MSFC  team  was  instructed  to  contact  the  UAH  team  members  if  any  questions  on 
interpretation  of  the  tools  came  up  during  the  review. 

The  phase  two  initial  assessment  of  the  tool  was  done  by  the  MSFC  senior  systems  engineer.  The 
SEPRT  and  SECRT  were  provided  along  with  the  assessment  methodology  and  supporting  documents. 
The  systems  engineer  did  have  experience  at  multiple  government  agencies,  but  focused  the  assessment 
based  on  the  tools  potential  use  at  NASA.  Each  of  the  tools  was  reviewed  individually  and  a  written 
summary  of  comments  and  questions  was  provided. 
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The  phase  three  detailed  assessment  was  conducted  by  the  senior  systems  engineer  and  senior  project 
manager.  An  additional  junior  systems  engineer  was  at  the  meeting  but  was  not  significantly  involved 
in  the  review.  The  results  of  the  initial  review  were  discussed  and  specific  questions  on  interpretation  of 
some  questions  were  answered.  The  MSFC  team  did  contact  the  UAH  team  for  clarification  on  some 
questions.  A  summary  of  the  findings  was  then  prepared. 

The  phase  four  reporting  of  the  results  was  done  by  submitting  the  summary  of  findings  and  conducting 
a  telephone  interview  to  discuss  findings  of  the  review.  The  web-based  survey  was  then  completed  and 
a  copy  of  the  results  of  the  survey  was  reviewed  by  the  team  members. 


3.  Results 

The  assessment  was  done  to  provide  an  external  assessment  of  the  potential  application  of  the  tools  in  a 
non-DoD  development  environment  and  to  provide  an  external  review  of  the  tools  by  product 
developers  working  in  a  similar  complex  system  environment.  The  SEPRT  and  SECRT  were  found  to 
be  generally  effective  in  supporting  an  aerospace  flight  hardware  program  at  NASA;  however, 
differences  in  the  models  used  for  project  development  and  terminology  differences  would  need  to  be 
addressed  before  the  model  would  be  applicable  for  general  use. 

The  SEPRT  and  SECRT  were  found  to  be  the  most  useful  in  concept  refinement,  system  development  & 
demonstration,  and  operations  &  support  phases.  The  tools  were  less  useful  in  the  technology 
development  and  production  &  deployment  phases.  The  assessment  team  noted  that  the  tools  could  be 
used  at  the  program,  project,  and  task  level.  Some  technology  development  activities,  such  as  science 
experiments,  may  not  find  the  tools  as  helpful  because  of  the  high  degree  of  process  tailoring.  The  team 
also  noted  that  there  were  differences  in  the  way  each  tool  presented  questions.  It  would  be  easier  for 
the  evaluators  if  the  tools  were  consistent  in  their  formats. 

The  evidence  required  to  complete  SEPRT  and  SECRT  was  generally  available.  It  was  noted  that  this 
could  be  highly  variable  and  is  dependent  in  part  on  the  experience  of  the  evaluators,  the  phase  the 
specific  program  is  in,  and  the  contracting  mechanisms  used.  Of  specific  note  was  the  experience  of  the 
evaluators  in  being  able  to  recognize  and  locate  relevant  artifacts  to  support  responses  to  specific 
questions  in  the  tools.  Accessing  artifacts  from  suppliers  could  be  of  concern  if  the  original  contracting 
documents  do  not  call  out  the  need  for  these. 

The  NASA  team  reported  approximately  five  labor  hours  was  required  by  their  team  members  to 
complete  each  of  the  tools.  It  was  noted  that  this  time  would  be  highly  variable,  dependent  on  the 
experience  of  the  assessment  team  members  and  the  phase  the  program  is  in.  Of  specific  concern  was 
the  need  to  adjust  the  tool  to  match  the  specific  uses  product  developer’s  model.  Having  an  external 
resource,  in  this  case  the  UAH  team,  to  provide  guidance  was  seen  as  valuable  to  the  effective  use  of  the 
tool. 

SEPRT  and  SECRT  were  reported  to  be  somewhat  effective  in  identifying  all  performance  risks.  The 
assessment  team  noted  that  it  can  be  difficult  to  isolate  the  performance  risk  portions  of  the  tool  from  the 
general  goals  and  questions,  and  nearly  all  the  questions  could  be  interpreted  as  having  an  effect  on 
performance  risk.  Understanding  the  purpose  of  the  tools  and  how  the  data  will  be  used  by  the  program 
management  team  was  noted  as  important  in  using  the  tools  effectively. 

The  team  reported  that  SEPRT  and  SECRT  were  assessed  to  be  somewhat  effective  in  helping  a 
program  team  use  an  evidence  based  approach  to  determining  performance  risks.  This  reported  result  is 
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of  note  because  the  tool  is  currently  structured  for  a  DoD  product  development  environment  and  does 
not  currently  have  tailoring  options  to  support  other  non-DoD  applications. 


4.  Lessons  Learned 

The  NASA  assessment  provided  several  lessons  and  opportunities  for  future  extensions  to  this  work. 
SEPRT  and  SECRT,  in  their  current  form,  are  targeted  for  DoD  MDAPs.  The  trade-off  is  that  the  more 
specific  the  tools  are  the  more  effective  they  can  potentially  be  in  managing  program  performance  risks. 
This  increased  effectiveness  is  offset  by  the  lack  of  a  broad  application  of  the  tools  to  other,  non-DoD 
applications.  One  extension  of  this  research  would  be  to  develop  a  tailoring  option  for  the  tools  so  that 
individual  programs  could  adjust  the  scope  of  the  tools  to  meet  their  specific  program  needs.  This 
tailoring  could  be  done  based  on  the  requirements  called  out  in  the  program  SEP  or  SEMP.  Another 
extension  would  be  to  develop  a  specific  application  set  for  DoD  and  a  second  more  generic  set  for  other 
applications. 

The  assessment  team  also  noted  the  tradeoff  between  having  an  experienced  but  heavily  tasked  senior 
assessment  team  complete  the  tools  versus  a  less  experienced  but  available  junior  team.  Senior 
managers  will  want  to  be  familiar  with  the  tools  if  they  are  going  to  implement  the  results.  Completion 
of  the  tool  would  most  likely  be  delegated  to  the  systems  engineering  team  or  the  logistics  team  to 
complete.  Having  an  experienced  evaluator  use  the  tools  help  with  accuracy  and  speed. 

To  increase  the  effectiveness  of  the  tool  the  NASA  assessment  team  noted  that  linking  the  SEPRT  and 
SECRT  to  specific  program  success  criteria  would  be  of  significant  value.  This  extension  to  the 
research  could  be  approached  in  two  ways.  The  first  would  be  to  link  the  tools  to  general  program 
success  criteria  such  as  technical  requirements,  budget  and  schedule.  These  would  be  generic  measures 
common  to  all  programs.  The  second  approach  would  be  to  link  the  tools  to  specific  technical  measures 
such  as  risk  metrics,  engineering  change  notices,  test  results,  and  requirements  stability.  This  second 
approach  would  be  a  significant  body  of  work,  but  would  ultimately  be  more  helpful  in  linking  the 
payback  of  investing  in  systems  engineering  to  specific  technical  performance  measures  rather  than 
general  programmatic  results. 


E.  SADB-DAPS  Based  SEPRT  and  SECRT  Evaluation  Summary 

A  study  was  conducted  to  determine  the  relationship  of  SEPRT ’s  system  engineering  effectiveness 
measures  with  the  existing  Department  of  Defense’s  Defense  Acquisition  Program  Support  (DAPS) 
Methodology  [22]  and  its  associated  Systemic  Analysis  Database  (SADB). 

In  general,  the  study  found  the  two  frameworks  to  be  complementary.  The  SEPRT  can  be  characterized 
of  as  a  subset  of  the  topics  covered  in  the  DAPS.  Initial  analysis  of  a  large  set  of  SADB  findings 
revealed  interesting  framework  comparison  details  and  indicates  an  opportunity  for  further  research. 
Supplementary  data  comparisons  and  analysis  of  additional,  relevant  SADB  findings  is  needed  to 
complete  the  analysis.  Study  results  indicate  that  the  two  frameworks  are  synergistic  and  may  be 
leveraged  as  complements  in  the  evaluation  of  MDAPS. 


1.  SEPRT  and  DAPS/SADB  Study  Objectives  and  Approach 

The  objectives  of  this  study  were  to: 
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•  Compare  SEPRT  with  DAPS  to  determine  gaps  and  coverage. 

•  Establish  a  baseline  for  revisions  and  future  comparisons. 

•  Provide  constructive  feedback  to  both  model  owners  as  appropriate. 

•  Determine  the  significance  of  SEPRT  effectiveness  measures  as  they  relate  to  SADB  negative 
findings. 

•  Understand  whether  it  is  possible  to  validate  the  set  of  SEPRT  systems  engineering  effectiveness 
measures  through  an  analysis  of  SADB  findings,  that  is  evaluate  the  ability  of  the  SEPRT  to  pre¬ 
identify  MDAP  problem  areas  articulated  in  the  SADB. 

Initial  research  focused  on  two  primary  activities:  (1)  mapping  DAPS  and  SEPRT  topical  areas  and  (2) 
analyzing  SEPRT-related  findings  in  the  SADB.  These  activities  were  performed  by  very  experienced 
subject  matter  experts  with  deep  system  development  and  federal  acquisition  experience;  however,  the 
tasks  were  labor  intensive  and  tedious  in  nature.  Consequently,  these  initial  results  are  not  exhaustive 
and  further  validation  and  verification  is  required. 

The  approach  used  to  map  the  DAPS  and  SEPRT  frameworks  included  identifying  and  selecting  the  key 
topical  areas,  searching  the  text  in  each  framework  using  key  words,  documenting  these  results  in  a 
Microsoft  Excel  spreadsheet,  and  analyzing  and  reviewing  the  resulting  comparison  map.  In  parallel,  an 
investigation  of  the  SADB  findings  was  performed  to  explore  the  use  of  the  SADB  in  evaluating  the 
ability  of  the  SEPRT  to  pre-identify  SADB  problem  areas.  This  included  submitting  a  request  for  SADB 
findings,  sorting  and  analyzing  the  data  set,  and  comparing  the  findings  with  the  SEPRT. 


2.  Existing  Defense  Acquisition  Program  Support  Tools 

The  Defense  Acquisition  Program  Support  (DAPS)  Methodology  was  developed  by  the  US  Department 
of  Defense,  Office  of  the  Deputy  Under  Secretary  of  Defense  for  Acquisition  and  Technology  (OUSD 
(AT&L)),  Systems  and  Software  Engineering.  The  most  recent  version,  2.0  (Change  3),  was  published 
March  20,  2009.  The  objectives  of  this  methodology  are  to: 

•  Improve  the  OUSD(AT&L)  decision-making  process  for  Major  Defense  Acquisition  Programs 
(MDAS)  and  Major  Automated  Information  Systems  programs  through  quality  systems  engineering 
and  program  assessment  support. 

•  Facilitate  successlul  execution  of  a  program  through  the  provision  of  independent,  actionable 
recommendations  to  the  government  program  management  office  (PMO). 

The  DAPS  Methodology  is  used  in  a  very  specific  Department  of  Defense  context.  In  this  context,  the 
methodology 

•  Provides  the  tailorable  framework  for  conducting  Program  Support  Reviews  (PSRs)  to  assist 
program  managers  and  DoD  decision  makers  in  preparation  for  milestone  decision  reviews. 

•  Provides  a  standardized  approach  (detailed  review  typology)  to  conducting  PSRs,  allowing  for  the 
participation  of  a  broad  cadre  of  subject  matter  experts  while  expecting  the  same  level  of  coverage 
and  quality  among  all  reviews. 
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•  Enabled  the  creation  of  the  Systemic  Analysis  Database  (SADB)  of  program  issues  and  root  causes. 
This  database  contains  findings  from  reviews  conducted  using  DAPS  and  allows  systemic  analyses 
that  can  be  used  to  effect  improvements  to  the  acquisition  process  (e.g.,  policies,  tools,  and 
education)  and  identify  best  practices. 

The  DAPS  Methodology  topical  area  content  focuses  on  systems  engineering,  but  covers  a  broader  range 
of  subjects  in  consideration  all  aspects  of  acquisition  management,  including  resource  planning, 
management  methods  and  tools,  earned  value  management,  logistics,  and  other  areas.  The  methodology 
is  composed  of  a  robust  listing  of  programmatic  and  technical  areas,  sub-areas,  and  factors.  It  was 
developed  to  be  both  broad  in  scope  and  detailed  enough  to  enable  application  to  programs  of  all  types. 

The  First-Level  Programmatic  and  Technical  Areas  are  defined  as  follows: 

1 .  Mission  Capabilities 

2.  Resources 

3.  Management 

4.  Technical  Process 

5.  Performance 

6.  Special  Interest  Areas 

The  DAPS  Methodology  provides  a  complete  description  of  each  programmatic  and  technical  area  as 
well  as  its  intended  use  and  processes. 


3.  SEPRT  -  DAPS  Framework  Mapping  Results 

Raw  data  results  of  the  mapping  between  the  SEPRT  and  DAPS  frameworks  were  documented  in  a 
Microsoft  Excel  spreadsheet.  Figure  1  provides  a  sample  of  this  mapping.  A  complete  mapping  of  the 
SEPRT  and  DAPS  frameworks  can  be  found  in  an  attachment  to  this  report.  As  indicated  previously, 
these  initial  results  are  not  exhaustive  and  further  validation  and  verification  is  required.  However,  it  can 
be  determined  that  the  two  frameworks  overlap  extensively  and  nearly  each  area  in  the  SEPRT  can  be 
traced  to  the  DAPS  in  some  fashion. 
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SE  EM  Framework  Area 

Interpretation 

DAPS 

Section 

DAPS  Topic  Covered 

Comments/Observations 

1  Concurrent  definition  of  system  requirements  and 
solutions 

1.1 

Understanding  of  stakeholder  needs:  capabilities, 
operational  concept,  key  performance  parameters, 
enterprise  fit  (legacv) 

1.1 

CONOPS 

1. 

1. 

1 

At  Milestone  A,  have  the  KPPs  been  identified  in  clear, 
comprehensive,  concise  terms  that  are  understandable 
to  the  users  of  the  system? 

Understandable, 

comprehensive 

requirements 

13.2 

KPPs  and  KSAs 

KPPs  are  required  to  be  "established  and 
documented";  no  guidance  on 
understandability/quality  of  KPP;  however 

states~> 

1. 

1. 

2 

Has  a  CONOPS  been  developed  showing  that  the 
system  can  be  operated  to  handle  both  nominal  and 
off-nominal  workloads,  and  to  meet  response  time 
requirements? 

feasible  workload 

demonstrated  at 

CONOPS;  are  scenarios 
quantified 

1.1 

1.2 

1.3.1 

CONOPS 

Analysis  of  Alternatives 

Reasonableness,  Stability,  and  Testability 

DAPS  specifies  "realistic"  scenarios,  not 
necessarily  "nominal  and  off-nominal 
workloads."  These  terms  and  "response 
time"  are  more  network-oriented. 
Quantification  is  given  by  "measures  of 
effectiveness." 

1. 

1. 

3 

Has  the  ability  of  the  system  to  meet  mission 
effectiveness  goals  been  verified  through  the  use  of 
modeling  and  simulation? 

verification  of  mission 
goals 

1.2.1 

1.3.1 

4.4.2 

Validity  and  Currency  (12  Analysis  of 
Alternatives) 

Reasonableness,  Stability,  and  Testability 
Modeling  and  Simulation  Tools 

Figure  10.  SEPRT  -  DAPS  Mapping  Sample 

Some  summary  observations  include: 

•  The  SEPRT  has  ‘rolled  up’  many  subtopics  into  fewer  effectiveness  areas;  DAPS  is  more  broad  and 
more  specific 

-  Specific  issues  (e.g.,  complementary  programs)  are  called  out  more  explicitly  in  DAPS 
and  handled  more  generically  as  ‘risks’  in  SEPRT. 

-  DAPS  discusses  DoD-specific  artifacts  and  products  (e.g.,  those  required  for  DoDAF, 
CARD,  JCIDS,  DIACAP). 

•  Several  key  concepts  found  in  SEPRT  seem  to  be  absent  or  used  in  a  different  manner  in  the  DAPS 
context,  such  as 

-  Negotiating  roles  &  responsibilities 

-  Nominal/off-nominal 

-  Stakeholder  concurrence/acceptance 

-  Validated 

-  Feasibility  evidence 

-  Timeboxing 

•  DAPS  is  very  DoD-process  oriented;  as  a  result,  its  guidance  on  what  the  program  should  do  when 
is  more  specific,  e.g.,  each  section  addresses  Milestones  A,  B,  and  C  as  appropriate. 

•  DAPS  has  more  information  about  expected  implementation,  often  providing  a  discussion  of  the 
rationale  or  importance  of  the  topic. 
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•  DAPS  makes  several  assumptions  regarding  the  programs’  required  characteristics,  processes,  such 
as  the  required  use  of  Modular  Open  Systems  Approach,  DoDAF,  and  Earned  Value  Management. 
These  types  of  assumptions  are  not  generally  found  in  SEPRT. 

Detailed  differences  between  the  frameworks  are  described  below: 

•  SEPRT  does  not  address  ‘ilities’  as  independent  topics  as  addressed  in  DAPS: 

-  5.2  Suitability 

■  5.2.1  Reliability  Assessment 

■  5.2.2  Availability  Assessment 

■  5.2.3  Maintainability  Assessment 

-  5.3  Survivability 

•  SEPRT  does  not  cover  domain  specifics  or  specialty  engineering  areas,  such  as  Security; 
Information  Assurance;  Weapons  Systems;  Spectrum  Management;  Human  Systems  Integration; 
Environment,  Safety,  Occupational  Health;  and  Corrosion. 

•  SEPRT  does  not  address  topics  beyond  the  development  phase,  including  production,  logistics, 
maintenance  upgrades. 

•  DAPS  often  mentions  an  incremental  approach,  but  it  does  not  address  how  to  divide  requirements 
into  increments  or  prioritize  requirements;  advocates  open  systems  (1.4,  1.4.4,  3.4.4). 

•  DAPS  requires  KPPs  be  established  and  documented,  but  there  is  no  guidance  on  the 
understandability/quality  of  KPP  (1.1.1). 

•  DAPS  does  not  explicitly  require  key  stakeholder  agreement/acceptance  on  the  system  boundary 
and  assumptions  of  its  environment  though  it  does  expect  that  collaboration  mechanisms  are  in 
place  (1.3.4,  2.3.1,  2.3.2,  2.4.1,  3.4.2,  4.5.4). 

•  DAPS  does  not  address  ‘negotiation’  of  roles  and  responsibilities,  but  rather  assumes  the  PM 
identifies  what  needs  to  be  done  and  who  shall  address  it  (1.1.4). 

•  DAPS  does  not  specifically  address  the  timeframe  of  personnel  assignments,  although  strategy  and 
schedule  realism  is  emphasized  (1.4.1,  2.1.3,  2.5.3). 

•  DAPS  does  address  quality  of  staff  in  general,  but  does  not  specifically  address  staff  needed  in 
critical  areas  or  what  constitutes  a  qualified  person  (2.1.1,  2.2.2). 

•  DAPS  has  an  Overarching  Integrated  Product  Team  but  it  is  unclear  on  this  group's  purpose  or 
participation  in  the  program  during  all  life  cycles;  a  “super  IPT”  is  not  mentioned,  but  DAPS  uses 
planning  and  reviews  to  resolve  issues  (2.2.3). 

•  DAPS  specifies  ‘realistic’  scenarios,  but  not  ‘nominal’  and  ‘off-nominal’  workloads  (1.1.2). 


36 


DAPS  is  not  specific  as  SEPRT  regarding  network-oriented  performance,  e.g.,  ‘response  times’ 

(1.1.2). 


•  DAPS  considers  flexibility  relative  to  system  design,  but  does  not  focus  on  the  ability  of 
requirements  to  take  on  future  mission  growth  over  the  program  lifecycle  (1.4.2). 

•  DAPS  seems  to  recommend  Modular  Open  Systems  Approach  as  an  approach  to  resolve  conflicts 
among  strongly  coupled  objectives  (2.3.3). 

•  DAPS  does  not  address  COTS  validation,  though  it  does  address  COTS  suitability  through  an 
acceptance  process  (1.2.4). 

•  DAPS  mentions  prototypes  as  part  of  the  SDD  process,  but  does  not  explicitly  mention  their  use  for 
mitigating  risks  (1.4.3). 

•  DAPS  addresses  competitive  prototyping  and  expects  the  results  to  be  reviewed  at  major  reviews, 
but  it  is  unclear  how  to  plan  for  it  from  a  contracting,  PM  perspective  (2.4.1,  2.4.2). 

•  DAPS  does  not  address  effective  strategies  for  addressing  proposed  changes,  such  as  triage  (4.4.1). 

•  DAPS  relates  the  expected  level  of  formality  with  size  of  the  project  in  terms  of  cost  versus  stability 
of  requirements  (4. 1.1). 

•  DAPS  does  not  appear  to  have  an  explicit  requirement  for  traceability  between  requirements  and 
architecture  (3.2.4). 

•  DAPS  does  not  address  the  concept  of  time-determined  development,  though  it  does  expect 
schedule  constraints  are  dealt  with  and  recommends  a  schedule  reserve  (3.4.4). 

•  DAPS  discusses  milestone  reviews  at  length;  however,  it  is  not  clear  that  evidence  of  feasibility  is 
available  and  ‘checking  the  box’  is  explicitly  avoided  (4.5). 

•  DAPS  does  not  explicitly  discuss  feasibility  evidence,  and  therefore  does  not  address  related 
progress  measures  (4.2.4). 

•  DAPS  discusses  milestone  reviews  at  length;  however,  it  is  not  clear  that  evidence  of  feasibility  is 
available  and  ‘checking  the  box’  is  explicitly  avoided  (4.5). 

It  should  be  noted  that  these  differences  are  not  described  as  positive  or  negative.  It  is  clear  that  the 
DAPS  Methodology  has  been  carefully  crafted  to  incorporate  the  constraints  and  complexities  of 
systems  development  within  the  DoD  context.  Discussions  with  the  DAPS  Methodology  owners  are 
required  to  determine  whether  there  is  opportunity  or  rationale  for  incorporating  SEPRT  key  concepts, 
for  example,  into  the  DAPS  and  conversely,  SEPRT  use  of  DAPS  elaborations. 


4.  SADB  Analysis  Results 

A  request  for  SADB  findings  was  requested  in  accordance  with  the  SADB  Report  Request  Form.  The 
data  requested  was  intended  to  be  related  to  SEPRT  Area  1:  Concurrent  Definition  of  System 
Requirements  and  Solutions.  The  following  DAPS  Sections  were  indicated  on  the  request  form: 
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1  Mission  Capabilities  (1.1  CONOPS,  1.2  Analysis  of  Alternatives,  1.3  Capabilities) 

2.2  Budget  Sufficiency  and  Phasing 

3.1  Acquisition  Strategy 

3.2  Knowledge-Based  Decisions  and  Milestones 

4.5.2  Verification  Correlation 

4.6  Design  Verification 

4.7  Supportability  Planning 

This  data  request  represents  9  of  59  DAPS  factors  for  comparison.  In  addition  to  the  checked  areas,  the 
request  included  the  following  additional  key  words:  Increments;  Understandable,  comprehensive, 
concise  requirements;  Verification  of  mission  goals;  Stakeholder  roles;  Legacy,  context,  operational 
concept  verification;  Exploration  of  alternative  solutions;  External  interfaces;  Third-party  solutions; 
Prioritization  of  requirements;  System  development. 

A  total  of  1,412  findings  were  received  in  response  to  this  request.  Of  these  findings,  704  were  classified 
as  Negative  Findings,  491  were  Neutral  Findings,  and  217  were  Positive  Findings.  Figure  11  illustrates 
the  characterization  of  the  data  set. 


Figure  11.  Data  Set 

Additional  data  characterization  observations  regarding  the  data  received  include: 

•  The  findings  received  represent  programs  sponsored  by  the  Army,  Navy,  Air  Force,  Marine  Corps, 
as  well  as  other  DoD  components. 

•  The  largest  group  of  findings  is  related  to  DAPS  section  1.3.1:  Reasonableness,  Stability, 
Testability. 

•  Of  ALL  findings  received,  nearly  half  or  more  are  related  to  each  DAPS  section  are  Negative 
Findings,  except  those  related  to  the  following  areas: 


Mission  Description 

1.2.1  Validity  and  Concurrency 
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-  1 .2.2  Linkage  and  Traceability 

-  4.7.2  Performance-Based  Logistics 
•  About  74%  of  the  NEGATIVE  findings 

-  Have  50  or  more  findings  related  to  its  DAPS  section 

-  Cover  the  following  seven  (7)  DAPS  areas: 

■  1.3.1  Reasonableness,  Stability,  Testability 

■  2.2.1  Program  Funding  and  Allocation 

■  Credibility 

■  Acceptability 

■  4.6.1  Test  and  Evaluation  Plan 

■  4.6.2  Verification  Correlation 

■  4.7.3  Sustainment 


5.  Recommendations  and  Conclusions 

The  initial  results  of  the  study  indicate  that  the  DAPS  and  SEPRT  frameworks  are  complementary. 
There  is  a  great  deal  of  overlap  between  the  frameworks  and  there  are  opportunities  to  leverage  details 
in  each  framework.  The  large  number  of  relevant  SADB  findings  (over  1,400)  indicates  synergy 
between  the  frameworks.  Specific  considerations  for  improvements  to  SEPRT  include: 

•  Clarify  the  program  life  cycle  assumed  by  the  SEPRT  framework,  for  example,  milestone  events 
and  timeline. 

•  Include  definitions  with  the  SEPRT  for  terms  not  currently  used,  e.g.,  mission  effectiveness; 
concurrent  solution;  feasibility  evidence;  quality  of  service;  program  governance  process;  level  of 
project  requirements  emergence;  earned  value  target. 

•  More  specific  guidance  is  needed  regarding  what  is  meant  by  ‘ validated ’  requirements,  quality  of 
service  guarantees,  and  solutions,  and  ‘ clearly  demonstrated  compliance’’  with  legal,  policy, 
regulatory,  standards,  security,  etc. 

•  Clarify  how  to  interpret  the  SEPRT  given  a  specific  perspective,  e.g.,  government  PMO  staffer 
versus  system  development  contractor. 

•  Provide  discussion  on  key  concepts,  especially  those  that  are  under-addressed  by  DAPS. 

•  Include  guidance  or  references  to  assist  in  understanding  and  implementation  of  the  SEPRT. 
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•  Differentiate  the  issues  cited  in  2.4.3:  resources,  process  infrastructure,  organizational  viability, 
project  alignment  with  business  unit. 

•  Separate  2.5.2  issues  related  to  empowerment  and  qualified  persons. 

•  Reword  3.2.1  to  be  more  system  oriented  versus  software. 

•  Clarify  meaning  of  3.2.5:  “Does  the  architecture  adequately  reconcile  functional  hardware  ‘part-of 
hierarchies  with  layered  software  ‘served-by’  hierarchies?” 

•  Change  earned  value  targets  to  earned  value  thresholds  in  4.2.4. 

As  mentioned  earlier,  it  is  recommended  that  discussions  with  DAPS  Methodology  owners  be  conducted 
to  provide  the  details  of  this  study  to  determine  if  there  are  any  salient  improvements  that  may  be  made 
to  the  DAPS. 

Additionally,  it  is  recommended  that  further  analysis  of  SADB  findings  be  conducted  to  leverage  the 
existing  data  and  refine  the  set  of  SEPRT  effectiveness  measures. 


F.  Conclusions  and  Recommendations 

1.  Conclusions 

DoD  programs  need  effective  systems  engineering  (SE)  to  succeed. 

DoD  program  managers  need  early  warning  of  any  risks  to  achieving  effective  SE. 

This  SERC  project  has  synthesized  the  best  analyses  of  DoD  SE  effectiveness  risk  sources  into  a  lean 
framework  and  toolset  for  early  identification  of  SE-related  program  risks. 

Three  important  points  need  to  be  made  about  these  risks. 

•  They  are  generally  not  indicators  of  “bad  SE.”  Although  SE  can  be  done  badly,  more  often  the 
risks  are  consequences  of  inadequate  program  funding  (SE  is  the  first  victim  of  an  underbudgeted 
program),  of  misguided  contract  provisions  (when  a  program  manager  is  faced  with  the  choice 
between  allocating  limited  SE  resources  toward  producing  contract-incentivized  functional 
specifications  vs.  addressing  key  performance  parameter  risks,  the  path  of  least  resistance  is  to  obey 
the  contract),  or  of  management  temptations  to  show  early  progress  on  the  easy  parts  while 
deferring  the  hard  parts  till  later. 

•  Analyses  have  shown  that  unaddressed  risk  generally  leads  to  serious  budget  and  schedule 
overruns. 

•  Risks  are  not  necessarily  bad.  If  an  early  capability  is  needed,  and  the  risky  solution  has  been 
shown  to  be  superior  to  the  alternatives,  accepting  and  focusing  on  mitigating  the  risk  is  generally 
better  than  waiting  for  a  better  alternative  to  show  up. 

The  results  of  the  SEPRT  and  SECRT  pilot  assessments,  the  DAPS  and  SADB  comparative  analysis, 
and  the  quantitative  business  case  analysis  for  the  use  of  the  SE  EM  framework,  tools,  and  operational 
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concepts  is  sufficiently  positive  to  conclude  that  implementation  of  the  approach  is  worth  pursuing. 
Presentations  at  recent  workshops  have  generated  considerable  interest  in  refining,  using,  and  extending 
the  capabilities  and  in  co-funding  the  followon  research.  However,  the  framework  and  prototype  tools 
have  been  shown  to  be  largely  efficacious  only  to  date  for  pilot  projects  done  by  familiar  experts  in  a 
relatively  short  time.  It  remains  to  demonstrate  how  well  the  framework  and  tools  will  perform  on  in- 
process  MDAPs  with  multiple  missions,  performers,  and  independent  expert  assessors. 

The  parametric  analysis  in  Section  A.3  concludes  that  the  greater  the  project’s  size,  criticality,  and 
stability  are,  the  greater  is  the  need  for  validated  architecture  feasibility  evidence  (i.e.,  evidence-based 
specifications  and  plans).  However,  for  very  small,  low-criticality  projects  with  high  volatility,  the 
evidence  generation  efforts  would  make  little  difference  and  would  need  to  be  continuously  redone, 
producing  a  negative  return  on  investment.  In  such  cases,  agile  methods  such  as  rapid  prototyping, 
Scrum  and  extreme  Programming  will  be  more  effective.  Overall,  evidence-based  specifications  and 
plans  will  not  guarantee  a  successful  project,  but  in  general  will  eliminate  many  of  the  software  delivery 
overruns  and  shortfalls  experienced  on  current  software  projects. 

Some  implications  of  defining  feasibility  evidence  as  a  “first  class”  project  deliverable  are  that  it  needs 
to  be  planned  (with  resources),  and  made  part  of  the  project’s  earned  value  management  system.  Any 
shortfalls  in  evidence  are  sources  of  uncertainty  and  risk,  and  should  be  covered  by  risk  management 
plans.  The  main  contributions  of  the  SERC  SE  EM  project  have  been  to  provide  experience-based 
approaches  and  operational  concepts  for  the  use  of  evidence  criteria,  evidence-generation  procedures, 
and  SE  effectiveness  measures  for  monitoring  evidence  generation,  which  support  the  ability  to  perform 
evidence-based  SE  on  DoD  MDAPs.  And  finally,  evidence-based  specifications  and  plans  such  as  those 
provided  by  the  SERC  SE  EM  capabilities  and  the  Feasibility  Evidence  Description  can  and  should  be 
added  to  traditional  milestone  reviews. 

As  a  bottom  line,  the  SERC  SE  capabilities  have  strong  potential  for  transforming  the  largely 
unmeasured  DoD  SE  activity  content  on  current  MDAPs  and  other  projects  into  an  evidence-based 
measurement  and  management  approach  for  both  improving  the  outcomes  of  current  projects,  and  for 
developing  a  knowledge  base  that  can  serve  as  a  basis  for  continuing  DoD  SE  effectiveness 
improvement. 


2.  Recommendations 

Based  on  the  Conclusions,  we  recommend  a  two-step  approach  for  achieving  a  SE  EM  initial 
operational  capability  and  transitioning  it  to  a  sustaining  organization.  Phase  Ha  is  proposed  to  begin 
with  research  on  three  tasks.  The  first  task  would  involve  experimentation  with  domain  extensions  and 
larger-scale  pilots.  The  second  task  would  involve  performing  and  analyzing  the  results  of  further 
completed  successful  and  unsuccessful  projects  to  test  the  hypothesis  that  there  is  a  critical  small  set  of 
critical  success-failure  factors  that  could  serve  as  top-level  early  warning  indicators.  The  third  task 
would  involve  extended  analyses  of  the  commonalities  and  variabilities  between  the  SERC  SE  EM 
apabilities  and  the  DAPS  methodology  and  SADB  results.  This  could  strengthen  both  and  enable  them 
to  be  used  in  complementary  ways. 

Phase  lib  would  continue  with  incremental  elaboration,  experimentation,  and  refinement  of  the  preferred 
approaches,  and  coordination  with  complementary  efforts.  Candidate  tasks  would  include  EM  tool  top- 
risk  summaries,  suggestions  for  mitigating  the  identified  risks,  ease  of  creating  domain-specific 
extensions,  creating  further  users-guide  and  tutorial  material,  creating  and  populating  a  knowledge  base 
of  the  results,  plans  for  transitioning  the  support  and  evolution  of  the  tools  to  an  appropriate  support 
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organization  such  as  DAU,  and  continuing  to  coordinate  the  tools’  content  with  complementary 
initiatives  such  as  the  INCOSE  Leading  Indicators  upgrade,  the  NDIA  enterprise-oriented  personnel 
competency  initiative,  and  the  SERC  SE  Body  of  Knowledge  and  Reference  Curriculum  RT. 
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COCOMO  -  Constructive  Cost  Model 

COSYSMO  -  Constructive  Systems  Engineering  Cost  Model 
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CSSE  -  Center  for  Systems  and  Software  Engineering 

CSF  -  Critical  Success  Factor 

DAPS  -  Defense  Acquisition  Program  Support 

DARPA  -  Defense  Advanced  Research  Projects  Agency 

DAU  -  Defense  Acquisition  University 

DCR  -  Development  Commitment  Review 
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DoD  -  Department  of  Defense 
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KPP  -  Key  Performance  Parameters 
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NDI  -  Non-Development  Item 
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OUSD(AT&L)  -  Office  of  the  Under  Secretary  of  Defense  (Acquisition,  Technology,  and  Logistics) 

PEO-PM  -  Program  Executive  Officer  -  Program  Manager 
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PSTL  -  Program  Support  Team  Leader 

RDT&E  -  Research,  Development,  Testing  &  Evaluation 
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SECRT  -  Systems  Engineering  Competency  Risk  Tool 
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SEPRT  -  Systems  Engineering  Performance  Risk  Tool 
SERC  -  Systems  Engineering  Research  Center 
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UML  -  Unified  Modeling  Language 
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APPENDIX  A:  GOALS,  CRITICAL  SUCCESS  FACTORS,  AND 

QUESTIONS 

This  section  has  the  Goals,  Critical  Success  Factors,  and  Questions  for  both  the  SEPRT  and  the  SECRT. 


SEPRT  -  Goals,  Critical  Success  Factors,  and  Questions 


Goal  1.  Concurrent  definition  of  system  requirements  and  solutions 

CSF  1.1  Understanding  of  stakeholder  needs:  capabilities,  operational  concept,  key 
performance  parameters,  enterprise  fit  (legacy) 

(a)  At  Milestone  A,  have  the  Key  Performance  Parameters  (KPPs)  been  identified  in  clear, 
comprehensive,  concise  terms  that  are  understandable  to  the  users  of  the  system? 

(b)  Has  a  Concept  of  Operations  (CONOPS)  been  developed  showing  that  the  system  can  be 
operated  to  handle  both  nominal  and  off-nominal  workloads  and  meet  response  time 
requirements? 

(c)  Has  the  ability  of  the  system  to  meet  mission  effectiveness  goals  been  verified  through  the  use 
of  modeling  and  simulation? 

(d)  Have  the  success-critical  stakeholders  been  identified  and  their  roles  and  responsibilities 
negotiated? 

(e)  Have  questions  about  the  fit  of  the  system  into  the  stakeholders'  context — acquirers,  end  users, 
administrators,  interoperators,  maintainers,  etc. — been  adequately  explored? 


CSF  1.2  Concurrent  exploration  of  solution  opportunities;  Analysis  of  Alternatives  (AoAs) 
for  cost-effectiveness  and  risk  (measures  of  effectiveness) 

(a)  Have  at  least  two  alternative  approaches  been  explored  and  evaluated? 

(b)  At  Milestone  B,  has  the  government  structured  the  program  plan  to  ensure  that  the  contractor 
addresses  the  allocation  of  capabilities  to  hardware,  software,  and  human  elements  sufficiently 
early  in  the  development  program? 

(c)  Has  the  claimed  degree  of  reuse  been  validated? 

(d)  Have  the  claimed  quality  of  service  guarantees  been  validated? 
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(e)  Have  proposed  Commercial  Off-the-Shelf  (COTS)  and  third-party  solutions  been  validated  for 
maturity,  compatibility,  supportability,  suitability,  and  effectiveness,  throughout  the  expected 
system  lifetime? 


CSF  1.3  System  scoping  &  requirements  definition  (external  interfaces;  Memoranda  of 
Agreement  (MoA)) 

(a)  Have  external  interface  complexities  been  identified  and  addressed  via  MoAs  or  their 
equivalent?  Is  there  a  plan  to  mitigate  their  risks? 

(b)  At  Milestone  B,  are  the  major  system-level  requirements  (including  all  KPPs)  defined 
sufficiently  to  provide  a  stable  basis  for  the  development  through  Initial  Operational  Capability 
(IOC)? 

(c)  By  Milestone  A,  is  there  a  plan  to  have  information  exchange  protocols  established  for  the  whole 
system  and  its  segments  by  Milestone  B? 

(d)  Have  the  key  stakeholders  agreed  on  the  system  boundary  and  assumptions  about  its 
environment? 


CSF  1.4  Prioritization  of  requirements  &  allocation  to  increments 

(a)  Can  an  initial  capability  be  achieved  within  the  time  that  the  key  program  leaders  are  expected  to 
remain  engaged  in  their  current  jobs  (normally  less  than  5  years  or  so  after  Milestone  B)?  If  this 
is  not  possible  for  a  complex  major  development  program,  can  critical  subsystems,  or  at  least  a 
key  subset  of  them,  be  demonstrated  within  that  time  frame? 

(b)  At  Milestone  B,  do  the  requirements  and  proposed  solutions  take  into  account  likely  future 
mission  growth  over  the  program  life  cycle? 

(c)  Have  appropriate  early  evaluation  phases,  such  as  competitive  prototyping,  been  considered  or 
executed  for  high-risk/low-maturity  components  of  the  system? 

(d)  Have  stakeholders  agreed  on  prioritization  of  system  features  and  their  allocation  to  development 
increments? 


Goal  2.  System  life-cycle  organization,  planning,  and  staffing 

CSF  2.1  Establishment  of  stakeholder  life-cycle  Responsibilities,  Authorities,  and 
Accountabilities  (RAAs)  (for  system  definition  &  system  development) 
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(a)  Are  the  stakeholders  who  have  been  identified  as  critical  to  the  success  of  the  project  represented 
by  highly  qualified  personnel  —  those  who  are  collaborative,  representative,  empowered, 
committed,  and  knowledgeable? 

(b)  At  Milestone  A,  are  there  validated  plans,  budgets,  and  schedules  defining  how  the  pre- 
Milestone  B  activity  will  be  done,  and  by  whom? 

(c)  Has  the  government  attempted  to  align  the  duration  of  the  program  manager's  assignment  with 
key  deliverables  and  milestones  in  the  program? 

(d)  Have  the  key  stakeholders  agreed  to  the  proposed  assignments  of  system  roles,  responsibilities, 
and  authorities? 


CSF  2.2  Establishment  of  Integrated  Product  Team  (IPT)  RAAs,  cross-IPT  coordination 
needs 

(a)  Does  the  project  make  effective  use  of  Integrated  Project  Teams  (IPTs)  throughout  the  supplier 
hierarchy? 

(b)  Are  the  IPTs  staffed  by  highly  qualified  personnel,  as  in  2.1  (a)? 

(c)  For  IPTs  addressing  strongly  coupled  objectives,  are  there  super-IPTs  for  resolving  conflicts 
among  the  objectives? 


CSF  2.3  Establishment  of  necessary  plans  and  resources  for  meeting  objectives 

(a)  Have  decisions  about  the  use  of  one-shot,  incremental,  or  evolutionary  development  been 
validated  for  appropriateness  and  feasibility,  and  accepted  by  the  key  stakeholders? 

(b)  Have  system  definition,  development,  test,  and  evolution  plans,  budgets,  and  schedules  been 
validated  for  appropriateness  and  feasibility,  and  accepted  by  the  key  stakeholders? 

(c)  Is  there  a  valid  business  case  for  the  system,  relating  the  life  cycle  system  benefits  to  the  system 
total  cost  of  ownership? 


CSF  2.4  Establishment  of  appropriate  source  selection,  contracting,  and  incentive  structures 

(a)  Has  the  competitive  prototyping  option  been  addressed,  and  the  decision  accepted  by  the  key 
stakeholders? 

(b)  If  doing  competitive  prototyping,  have  adequate  plans  and  preparations  been  made  for  exercising 
and  evaluating  the  prototypes,  and  for  sustaining  core  competitive  teams  during  evaluation  and 
downselecting? 
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(c)  Is  the  status  of  the  candidate  performer's  business  and  team  "healthy,"  both  in  terms  of  business 
indicators,  and  within  the  industrial  base  for  the  program  area?  Is  the  program  aligned  with  the 
core  business  of  the  unit,  and  staffed  adequately  and  appropriately? 

(d)  Has  the  acquiring  organization  successfully  completed  projects  similar  to  this  one  in  the  past? 

(e)  Has  the  candidate  performing  organization  successfully  completed  projects  similar  to  this  one  in 
the  past? 

(f)  Is  the  program  governance  process,  and  in  particular  the  system  engineering  plan,  well 
articulated  and  compatible  with  the  goals  of  the  program? 


CSF  2.5  Establishment  of  necessary  personnel  competencies 

(a)  Does  the  government  have  access  over  the  life  of  the  program  to  the  talent  required  to  manage 
the  program?  Does  it  have  a  strategy  over  the  life  of  the  program  for  using  the  best  people 
available  in  the  government,  the  Federally  Funded  Research  and  Development  Centers 
(FFRDCs),  and  the  professional  service  industry? 

(b)  At  Milestone  B,  have  sufficiently  talented  and  experienced  program  and  systems  engineering 
managers  been  identified?  Have  they  been  empowered  to  tailor  processes  and  to  enforce 
development  stability  from  Milestone  B  through  IOC? 

(c)  Has  the  government  attempted  to  align  the  duration  of  the  program  manager's  assignment  with 
key  deliverables  and  milestones  in  the  program? 

(d)  Is  the  quantity  of  developer  systems  engineering  personnel  assigned,  their  skill  and  seniority 
mix,  and  the  time  phasing  of  their  application  throughout  the  program  lifecycle,  appropriate? 


Goal  3.  Technology  Maturing,  Architecting 

CSF  3.1  COTS/Non-Development  Item  (NDI)/Services  evaluation,  selection,  validation  for 
maturity  &  compatibility 

(a)  Have  COTS/NDI/Services  opportunities  been  evaluated  prior  to  baselining  requirements? 

(b)  Have  COTS/NDI/Services  scalability,  compatibility,  quality  of  service,  and  life  cycle  support 
risks  been  thoroughly  addressed? 

(c)  Has  a  COTS/NDI/Services  life  cycle  refresh  strategy  been  developed  and  validated? 


CSF  3.2  Life-cycle  architecture  definition  &  validation 
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(a)  Has  the  system  been  partitioned  to  define  segments  that  can  be  independently  developed  and 
tested  to  the  greatest  degree  possible? 

(b)  By  Milestone  A,  is  there  a  plan  to  have  internal  and  external  information  exchange  protocols 
established  and  validated  for  the  whole  system  and  its  segments  by  Milestone  B? 

(c)  Does  the  project  have  adequate  processes  in  place  to  define  the  verification,  test  &  validation, 
and  acceptance  of  systems  and  system  elements  at  all  phases  of  definition  and  development? 

(d)  Is  there  a  clear,  consistent,  and  traceable  relationship  between  system  requirements  and 
architectural  elements?  Have  potential  off-nominal  architecture-breakers  been  addressed? 

(e)  Does  the  architecture  adequately  reconcile  functional  hardware  part-of  hierarchies  with  layered 
software  served-by  hierarchies? 

(f)  Has  a  Work  Breakdown  Structure  (WBS)  been  developed  with  the  active  participation  of  all 
relevant  stakeholders,  which  accurately  reflects  both  the  hardware  and  the  software  product 
structure? 


CSF  3.3  Use  of  prototypes,  exercises,  models,  and  simulations  to  determine  technological 
solution  maturity 

(a)  Will  risky  new  technology  mature  before  Milestone  B?  Is  there  a  risk  mitigation  plan? 

(b)  Have  the  key  non-technical  risk  drivers  been  identified  and  covered  by  risk  mitigation  plans? 

(c)  Is  there  a  sufficient  collection  of  models  and  appropriate  simulation  and  exercise  environments 
to  validate  the  selected  concept  and  the  CONOPS  against  the  KPPs? 

(d)  Has  the  claimed  degree  of  reuse  been  validated? 


CSF  3.4  Validated  system  engineering,  development,  manufacturing,  operations  & 
maintenance  budgets  and  schedules 

(a)  Are  the  major  known  cost  and  schedule  drivers  and  risks  explicitly  identified,  and  is  there  a  plan 
to  track  and  reduce  uncertainty? 

(b)  Have  the  cost  confidence  levels  been  developed  and  accepted  by  the  key  system  stakeholders? 

(c)  Is  there  a  top-to-bottom  plan  for  how  the  total  system  will  be  integrated  and  tested?  Does  it 
adequately  consider  integration  facilities  development  and  earlier  integration  testing? 

(d)  If  timeboxing  or  time-determined  development  is  used  to  stabilize  schedules,  have  features  been 
prioritized  and  the  system  architected  for  ease  of  adding  or  dropping  borderline  features? 
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(e)  Are  there  strategies  and  plans  for  evolving  the  architecture  while  stabilizing  development  and 
providing  continuity  of  service? 


Goal  4.  Evidence-based  progress  monitoring  and  commitment  reviews 

CSF  4.1  Monitoring  of  system  definition,  development  and  test  progress  vs.  plans 

(a)  Are  the  levels  and  formality  of  plans,  metrics,  evaluation  criteria,  and  associated  mechanisms 
(IMP,  IMS,  WBS,  EVMS)  commensurate  with  the  level  of  project  requirements  emergence  and 
stability?  (too  little  is  risky  for  pre-specifiable  and  stable  requirements;  too  much  is  risky  for 
emergent  and  unstable  requirements) 

(b)  Are  the  project's  staffing  plans  and  buildup  for  progress  monitoring  adequate  with  respect  to 
required  levels  of  expertise? 

(c)  Have  most  of  the  planned  project  personnel  billets  been  fdled  with  staff  possessing  at  least  the 
required  qualification  level? 

(d)  Is  the  project  adequately  identifying  and  managing  its  risks? 

(e)  Have  the  processes  for  conducting  reviews  been  evaluated  for  feasibility,  reasonableness, 
completeness,  and  assurance  of  independence? 

(f)  Has  compliance  with  legal,  policy,  regulatory,  standards,  and  security  requirements  been  clearly 
demonstrated? 


CSF  4.2  Monitoring  of  feasibility  evidence  development  progress  vs.  plans 

(a)  Has  the  project  identified  the  highest  risk  areas  on  which  to  focus  feasibility  analysis? 

(b)  Has  the  project  analyzed  alternative  methods  of  evaluating  feasibility  (models,  simulations, 
benchmarks,  prototypes,  reference  checking,  past  performance,  etc.)  and  prepared  the 
infrastructure  for  using  the  most  cost-effective  choices? 

(c)  Has  the  project  identified  a  full  set  of  representative  operational  scenarios  across  which  to 
evaluate  feasibility? 

(d)  Has  the  project  prepared  milestone  plans  and  earned  value  targets  for  measuring  progress  in 
developing  feasibility  evidence? 

(e)  Is  the  project  successfully  monitoring  progress  and  applying  corrective  action  where  necessary? 
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CSF  4.3  Monitoring,  assessment,  and  replanning  for  changes  in  needs,  opportunities,  and 
resources 

(a)  Does  the  project  have  an  effective  strategy  for  performing  triage  (accept,  defer,  reject)  on 
proposed  changes,  that  does  not  destabilize  ongoing  development? 

(b)  Does  the  project  have  an  adequate  capability  for  performing  change  impact  analysis  and 
involving  appropriate  stakeholders  in  addressing  and  prioritizing  changes? 

(c)  Is  the  project  adequately  verifying  and  validating  proposed  changes  for  feasibility  and  cost- 
effectiveness? 


CSF  4.3  Use  of  milestone  reviews  to  ensure  stakeholder  commitment  to  proceed? 

(a)  Are  milestone  review  dates  based  on  availability  of  feasibility  evidence  versus  on  availability  of 
artifacts  or  on  planned  review  dates? 

(b)  Are  artifacts  and  evidence  of  feasibility  evaluated  and  risky  shortfalls  identified  by  key 
stakeholders  and  independent  experts  prior  to  review  events? 

(c)  Are  developer  responses  to  identified  risks  prepared  prior  to  review  events? 

(d)  Do  reviews  achieve  risk-based  concurrence  of  key  stakeholders  on  whether  to  proceed  into  the 
next  phase?  (proceed;  skip  a  phase;  revisit  the  current  phase;  terminate  or  rescope  the  project) 
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SECRT  -  Goals,  Critical  Success  Factors,  and  Questions 


Goal  1.  Concurrent  definition  of  system  requirements  and  solutions 

CSF  1.1  Understanding  of  stakeholder  needs:  capabilities,  operational  concept,  key 

performance  parameters,  enterprise  fit  (legacy).  Ability  to  analyze  strengths  and 
shortfalls  in  current-system  operations  via: 

(a)  Participatory  workshops,  surveys,  focus  groups 

(b)  Operations  research  techniques:  operations  data  collection  and  analysis 

(c)  Mission  effectiveness  modeling  and  simulation 

(d)  Prototypes,  scenarios,  stories,  personas 

(e)  Ethnographic  techniques:  Interviews,  sampled  observations,  cognitive  task  analysis 


CSF  1.2  Concurrent  exploration  of  solution  opportunities;  Analysis  of  Alternatives  for  cost- 
effectiveness  &  risk  (Measures  of  Effectiveness).  Ability  to  identify  and  assess 
alternative  solution  opportunities  via  experimentation  and  analysis  of: 

(a)  Alternative  work  procedures,  non-materiel  solutions 

(b)  Purchased  or  furnished  products  and  services 

(c)  Emerging  technology 

(d)  Competitive  prototyping 


CSF  1.3  System  scoping  &  requirements  definition  (External  interfaces;  Memoranda  of 
Agreement).  Ability  to  establish  system  scope  and  requirements  via: 

(a)  Cost-schedule-effectiveness  assessment  of  needs  vs.  opportunities 

(b)  Organizational  responsibilities,  authorities,  and  accountabilities  (RAAs)  assessment 

(c)  Appropriate  degrees  of  requirements  completeness,  consistency,  testability,  and  variability  due 
to  emergence  considerations 
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CSF  1.4 


Prioritization  of  requirements  and  scheduling  into  increments.  Ability  to  prioritize 
requirements  and  schedule  them  into  increments  based  on  considerations  of: 


(a)  Stakeholder  priorities  and  returns  on  investment 

(b)  Capability  interdependencies  and  requirements  emergence  considerations 

(c)  Technology  maturity  and  implementation  feasibility  risks 

Goal  2.  System  Life  Cycle  Organization,  Planning,  Staffing 

CSF  2.1  Establishment  of  stakeholder  life  cycle  RAAs  for  system  definition,  system 

development,  and  system  operation.  Ability  to  support  establishment  of  stakeholder 
RAAs  via  conduct  of: 

(a)  Organizational  capability  analyses 

(b)  Stakeholder  negotiations 

(c)  Operational  exercise  analyses 

CSF  2.2  Establishment  of  Integrated  Product  Team  (IPT)  RAAs,  Cross-IPT  coordination 
needs.  Ability  to  establish  IPT  RAAs  and  cross-IPT  coordination  mechanisms  via: 

(a)  Risk  identification,  analysis,  and  prioritization 

(b)  Organizational  RAAs  and  skills  availability  assessment 

(c)  Risk  interdependency  analysis 

(d)  Risk  resolution  cost-benefit  analysis 


CSF  2.3  Establishment  of  necessary  resources  for  meeting  objectives.  Ability  to  support 
program  negotiation  of  objectives  vs.  resources  via: 

(a)  Cost-schedule-capability  tradeoff  analyses 

(b)  Use  of  requirements  priorities  and  interdependencies  to  support  negotiation  of  increment 
contents 

(c)  Development  of  strategies  to  adjust  increment  content  to  meet  delivery  schedules 

(d)  Analysis  of  project  change  traffic  and  rebaselining  of  future-increment  plans  and  specifications 
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CSF  2.4 


Establishment  and  usage  support  of  appropriate  source  selection,  contracting,  & 
incentive  structures.  Ability  to  support  program  management  in  preparing  source 
selection  materials,  matching  contracting  and  incentive  structures  to  program 
objectives,  and  technical  monitoring  of  performance  vs.  program  objectives: 

(a)  Preparation  of  proposal  solicitation  materials  and  evaluation  capabilities  and  procedures 

(b)  Evaluation  of  proposal  submissions  with  respect  to  criteria 

(c)  Technical  support  of  contract  negotiations 

(d)  Technical  support  of  contract  performance  monitoring 

CSF  2.5  Assurance  of  necessary  personnel  competencies.  Ability  to  support  program 
management  in  evaluating  proposed  staffing  plans  and  monitoring  staffing 
capabilities  vs.  plans  in  the  areas  of: 

(e)  Concurrent  Definition  of  System  Requirements  &  Solutions 

(f)  System  Life  Cycle  Organization,  Planning,  Staffing 

(g)  Technology  Maturing  and  Architecting 

(h)  Evidence-Based  Progress  Monitoring  &  Commitment  Reviews 

(i)  Professional  and  Interpersonal  Skills 


Goal  3.  Technology  Maturing  and  Architecting 

CSF  3.1  COTS/NDI/Services  evaluation,  selection,  validation  for  capability,  maturity  & 
compatibility.  Ability  to  evaluate  alternative  combinations  of  COTS,  NDI,  and 
purchased  services  for: 

(a)  Functional  capabilities  vs.  system  needs 

(b)  Levels  of  service:  performance,  resilience,  scalability,  usability,  tailorability,  etc. 

(c)  Mutual  compatibility  and  external  interoperability 

(d)  Supplier  maturity,  stability,  support,  and  responsiveness 

(e)  Acquisition  and  operational  costs 
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CSF  3.2 


Life  Cycle  architecture  definition  &  validation.  Ability  to  define  and  evolve 
configurations  of  hardware  and  software  components  and  connectors  along  with 
human  operational  architectures,  and  to  validate  that  they  cost-effectively  support 
program  objectives: 

(a)  Define  candidate  hardware/software/human-operational  architectures 

(b)  Evaluate  their  functional  capabilities,  levels  of  service,  interoperability,  and  sustainability  vs. 
system  needs 

(c)  Perform  tradeoff  analyses  among  functional  capabilities  and  levels  of  service 


CSF  3.3  Use  of  prototypes,  exercises,  models,  and  simulations  to  determine  technology 

maturity,  architecture  feasibility.  Ability  to  assess  the  relative  costs  and  benefits  of 
alternative  evaluation  methods,  and  apply  appropriate  combinations  of  methods: 

(a)  Assess  relative  costs,  schedules,  and  capabilities  of  various  combinations  of  evaluation  methods 

(b)  Prepare  plans  for  enabling  and  performing  evaluations 

(c)  Prepare  representative  nominal  and  off-nominal  scenarios,  workload  generators,  virtual 
component  surrogates,  and  testbeds  to  support  evaluations 

(d)  Perform  evaluations,  analyze  results,  investigate  anomalies,  and  adjust  plans  as  appropriate 


CSF  3.4  Validated  System  Engineering,  Development,  Manufacturing,  Operations  & 
Maintenance  budgets  &  schedules.  Ability  to: 

(a)  Assess  alternative  budget  and  schedule  estimation  methods  vs.  nature  of  system,  degree  of 
system  knowledge,  complementarity  of  estimates,  and  cost  vs.  accuracy  of  performing  estimates 

(b)  Prepare  plans  for  gathering  inputs  and  performing  estimates 

(c)  Perform  selected  combinations  of  estimates  and  reconcile  their  differences 

(d)  Perform  tradeoff  analyses  among  functional  capabilities,  levels  of  service,  costs,  and  schedules 


Goal  4.  Evidence-Based  Progress  Monitoring  &  Commitment  Reviews 

CSF  4.1  Monitoring  of  system  definition,  development,  and  test  progress  vs.  plans.  Ability  to 
plan,  monitor,  and  evaluate  technical  progress  vs.  plans 
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(a)  Prepare  test  and  evaluation  facilities  and  plans,  and  define  data  to  be  provided  for  assessing 
technical  progress  vs.  project  plans 

(b)  Monitor  performers’  technical  progress  in  developing,  verifying,  and  validating  their  technical 
solutions 

(c)  Identify  shortfalls  in  technical  progress  vs.  plans,  and  determine  their  root  causes 


CSF  4.2  Monitoring  of  feasibility  evidence  development  and  test  progress  vs.  plans.  Ability 
to: 

(a)  Evaluate  developers’  feasibility  evidence  assessment  and  test  plans  for  coverage,  cost- 
effectiveness,  and  realism  of  assumptions 

(b)  Monitor  developers’  progress  with  respect  to  plans,  identify  shortfalls  and  root  causes 

(c)  Evaluate  feasibility  evidence  produced,  identify  shortfalls  and  root  causes 


CSF  4.3  Monitoring,  assessment,  and  replanning  for  changes  in  needs,  opportunities,  and 
resources.  Ability  to: 

(a)  Assess  proposed  changes  in  program  objectives,  constraints,  plans,  and  resources 

(b)  Perform  triage  to  determine  which  changes  should  be  handled  immediately,  deferred  to  future 
increments,  or  rejected 

(c)  Perform  tradeoff  analyses  to  support  renegotiation  of  current  and  future  increment  plans  and 
contents 

(d)  Validate  feasibility  and  cost-effectiveness  of  renegotiated  increment  plans  and  contents 

(e)  Monitor  effectiveness  of  configuration  and  version  management 


CSF  4.4  Identification  and  mitigation  planning  for  feasibility  evidence  shortfalls  and  other 

risks.  Ability  to  recommend  corrective  actions  for  feasibility  evidence  shortfalls  and 
other  risks 

(a)  Identify  and  evaluate  alternative  courses  of  action  to  address  feasibility  evidence  shortfalls, 
technical  risks,  and  root  causes 

(b)  Recommend  appropriate  corrective  actions  to  obtain  best-possible  system  outcomes 
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CSF  4.5  Use  of  milestone  reviews  to  ensure  stakeholder  commitment  to  proceed.  Ability  to: 

(a)  Prepare  plans,  schedules,  budgets,  scenarios,  and  facilities  for  evaluating  developer  feasibility 
evidence 

(b)  Identify  shortfalls  in  feasibility  evidence  as  program  risks 

(c)  Assess  developer  risk  management  plans  for  coverage  of  risks,  identify  shortfalls,  and 
recommend  corrective  actions 


Goal  5.  Professional  and  Interpersonal  Skills 

CSF  5.1  Leadership.  Ability  to  plan,  staff,  organize,  teambuild,  control,  and  direct  systems 
engineering  teams 

(a)  Prepare  top-level  plans,  schedules,  budgets,  and  deliverables  for  a  system  engineering  team 

(b)  Evaluate  and  recruit  appropriate  staff  members  for  executing  plans 

(c)  Involve  staff  members  in  collaborative  development  of  team  shared  vision,  detailed  plans,  and 
organizational  roles;  adjust  top-level  plans  as  appropriate 

(d)  Monitor  progress  with  respect  to  plans,  identify  shortfalls,  provide  mentoring  and  constructive 
corrective  actions  to  address  shortfalls 


CSF  5.2  Collaboration.  Ability  to  work  with  others  to  negotiate,  plan,  execute,  and 
coordinate  complementary  tasks  for  achieving  program  objectives 

(a)  Develop  understanding  of  other  participants’  value  propositions,  and  use  knowledge  to  negotiate 
mutually  satisfactory  roles,  responsibilities,  and  modes  of  collaboration 

(b)  Establish  modes  of  pro-active  coordination  of  emerging  issues  with  other  team  members  and 
teams 

(c)  Provide  help  to  others  in  need  of  your  capabilities 


CSF  5.3  Communication.  Ability  to  perform  timely,  coherent,  and  concise  verbal  and 
written  communication 

(a)  Develop  understanding  of  other  participants’  knowledge  boundaries  and  terminology,  and  adjust 
your  terminology  to  facilitate  their  understanding  of  your  communications 
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(b)  Provide  timely,  coherent,  and  concise  verbal  and  written  communication  within  your  team  and 
among  external  stakeholders 

(c)  Explore  new  low-tech  (wallboards)  and  high-tech  (wikis,  blogs,  videos)  modes  of  effective 
communications 


CSF  5.4  Accountability.  Ability  to  deliver  on  promises  and  behave  ethically 

(a)  Commit  to  and  follow  through  on  promised  commitments;  provide  advance  warning  of  potential 
delays  and  shortfalls 

(b)  Respect  the  truth,  intellectual  property,  and  the  rights  and  concerns  of  others 


CSF  5.5  Adaptability  and  Leaning.  Ability  to  cope  with  uncertainty  and  unexpected 
developments,  and  to  seek  help  and  fill  relevant  knowledge  gaps 

(a)  Be  prepared  to  cope  with  inevitable  uncertainty  and  unexpected  developments 

(b)  Identify  key  knowledge  and  skills  needed  for  your  project  and  career,  and  engage  in  learning 
activities  to  master  them 
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APPENDIX  B:  SEPRT  EXAMPLE  -  LOGISTICS  SUPPORT  SYSTEM 

PROJECT 
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NUTh:  impact  and  evidence/risk  ratings  should  be  done  independently. 
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Exposure 

Source  of  evidence 

Goal  1: 


Concurrent  definition  of  system  requirements  and  solutions 


Critical  Success  Factor  1.1 


Understanding  of  stakeholder  needs:  capabilities,  operational 
concept,  key  performance  parameters,  enterprise  fit  (legacy) 


At  Milestone  A,  have  the  KPPs  been  identified  in  clear,  comprehensive, 
concise  terms  that  are  understandable  to  the  users  of  the  system? 


No  formal  Milestone  A 


Has  a  CONOPS  been  developed  showing  that  the  system  can  be  operated 
to  handle  both  nominal  and  off-nominal  workloads,  and  to  meet  response 
time  requirements? 

Has  the  ability  of  the  system  to  meet  mission  effectiveness  goals  been 
verified  through  the  use  of  modeling  and  simulation? 

Have  the  success-critical  stakeholders  been  identified  and  their  roles  and 
responsibilities  negotiated? 


Have  questions  about  the  fit  of  the  system  into  the  stakeholders'  context  -- 
acquirers,  end  users,  administrators,  interoperators,  maintainers,  etc.  -- 
been  adequately  explored? 


IT  system  sized  using  vendor  benchmarks  and  expected 
number  of  users 

IT  system  designed  to  replace  legacy  system  and  manual 
processes.  Mission  effectiveness  of  system  not  a  major 
concern  during  development. 

Development  of  system  had  been  attempted  by  other 
companies  and  failed.  Stakeholders  had  been  previous 
identified  and  were  involved  early  on. 

Explored  across  all  areas  early  on.  However,  new  sponsor 
IT  PM  (assigned  to  project  after  system  acceptance  but 
before  deployment)  changed  system  requirements  related 
to  database  system  and  there  was  no  funding  left  to 
migrate  to  a  different  DBMS. 


Critical  Success  Factor  1.2 


Concurrent  exploration  of  solution  opportunities;  analysis  of 
alternatives  (AoAs)  for  cost-effectiveness  and  risk  (measures  of 
effectiveness) 

Have  at  least  two  alternative  approaches  been  explored  and  evaluated? 

At  Milestone  B,  has  the  government  structured  the  program  plan  to 
ensure  that  the  contractor  addresses  the  allocation  of  capabilities  to 
hardware,  software,  and  human  elements  sufficiently  early  in  the 
development  program? 

Has  the  claimed  degree  of  reuse  been  validated? 


Was  recommended,  but  rejected  by  sponsor  PM  due  to 
"color  of  money"  (legacy  replacement  needed  to  remain  on 
upgraded  legacy  platform) 


No  planned  reuse  (unless  you  consider  use  of  GUI  builder 
"reuse".  In  this  case,  none  initially  planned,  but  change  in 
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Critical  Success  Factor  1.3 


2 


2 


1 


Critical  Success  Factor  1.4 


Have  the  claimed  quality  of  service  guarantees  been  validated? 

Have  proposed  COTS  and  third-party  solutions  been  validated  for 
maturity,  compatibility,  supportability,  suitability,  and  effectiveness, 
throughout  the  expected  system  lifetime? 

System  scoping  &  requirements  definition  (external  interfaces; 
memoranda  of  agreement) 

Have  external  interface  complexities  been  identified  and  minimized  via 
MoAs  or  their  equivalent?  Is  there  a  plan  to  mitigate  their  risks? 

At  Milestone  B,  are  the  major  system-level  requirements  (including  all 
KPPs)  defined  sufficiently  to  provide  a  stable  basis  for  the  development 
through  IOC? 

By  Milestone  A,  is  there  a  plan  to  have  information  exchange  protocols 
established  for  the  whole  system  and  its  segments  by  Milestone  B? 

Have  the  key  stakeholders  agreed  on  the  system  boundary  and 
assumptions  about  its  environment? 

Prioritization  of  requirements  &  allocation  to  increments 

Can  an  initial  capability  be  achieved  within  the  time  that  the  key  program 
leaders  are  expected  to  remain  engaged  in  their  current  jobs  (normally 
less  than  5  years  or  so  after  Milestone  B)?  If  this  is  not  possible  for  a 
complex  major  development  program,  can  critical  subsystems,  or  at  least 
a  key  subset  of  them,  be  demonstrated  within  that  time  frame? 


At  Milestone  B,  do  the  requirements  take  into  account  likely  future 
mission  growth  over  the  program  life  cycle? 


Have  appropriate  early  evaluation  phases,  such  as  competitive 
prototyping,  been  considered  or  executed  for  high-risk/low-maturity 
components  of  the  system? 

Have  stakeholders  agreed  on  prioritization  of  system  features  and  their 
allocation  to  development  increments? 


Vendor  platform  benchmarks  used 

GUI  prototype  developed  and  demo'd  prior  to  finalizing  the 
decision  to  use  the  GUI  builder. 

Not  clear  why  this  was  yellow  based  on  response  to  first 
subitem  (High  Impact,  Good  Evidence)--Also  stayed  yellow 
after  response  to  2nd  subitem 

External  interfaces  well  defined  and  used  by  legacy  system 
being  replaced. 


N/A--no  formal  Milestone  A  and  system  used  well-defined, 
existing  protocols  for  external  interfaces. 

Based  on  current  business  processes  and  documented  in 
system  requirements 


This  was  not  initially  perceived  as  a  high  impact,  but  turned 
out  to  be  a  high  impact.  System  was  developed,  accepted, 
and  entered  OT&E  under  a  single  program  leader,  but 
leader  was  replaced  prior  to  deployment  and  subsequent 
leader  decided  that  she  did  not  want  to  deploy  the  system 
since  it  did  not  use  her  "preferred"  DBMS. 

Some  initial  problems  during  OT&E  due  to  the  fact  that 
developers  had  not  anticipated  how  often  users  would 
resend  transactions  and  users  had  request  that  ALL 
transactions  be  saved  online.  Quick  fix  initiated  during 
OT&E  when  storage  capacity  of  system  exceeded. 


Single  increment  on  fixed  price  contract. 
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Goal  2: 


System  life-cycle  organization,  planning,  and  staffing 


Critical  Success  Factor  2.1 


Establishment  of  stakeholder  life-cycle  responsibilities,  authorities, 
and  accountabilities  (RAAs)  (for  system  definition  &  system 
development) 

Are  the  stakeholders  who  have  been  identified  as  critical  to  the  success  of 
the  project  represented  by  highly  qualified  personnel  --  those  who  are 
collaborative,  representative,  authorized,  committed,  and  knowledgeable? 


System  developed  for  small  organization  and  all  key  users 
and  associated  managers  identified  as  stakeholders  and 
actively  participated  in  the  development  process. 


At  Milestone  A,  are  there  validated  plans,  budgets,  and  schedules  defining 
how  the  pre-Milestone  B  activity  will  be  done,  and  by  whom? 

Has  the  government  attempted  to  align  the  duration  of  the  program 
manager's  assignment  with  key  deliverables  and  milestones  in  the 
program? 

Have  the  key  stakeholders  agreed  to  the  proposed  assignments  of  system 
roles,  responsibilities,  and  authorities? 


No  formal  Milestone  A 


PM  lasted  through  acceptance  testing  and  into  OT&E 

Initial  assessed  impact  low,  but  turned  out  to  be  high  when 
users  realized  that  their  job  was  at  risk  since  the  new 
system  automated  much  of  their  job,  resulting  in  them 
sabotaging  the  deployment  of  the  system  (along  with  the 
new  PM). 


Critical  Success  Factor  2.2 


Establishment  of  integrated  product  team  (IPT)  RAAs,  cross-IPT 
coordination  needs 

Does  the  project  make  effective  use  of  Integrated  Project  Teams  (IPTs) 
throughout  the  supplier  hierarchy? 


1 


N/A--small  agile-like  team 


Are  the  IPTs  staffed  by  highly  qualified  personnel,  as  in  2.a(a)? 


N/A 


For  IPTs  addressing  strongly  coupled  objectives,  are  there  "super-IPTs"  for 
resolving  conflicts  among  the  objectives? 


Critical  Success  Factor  2.3 


Establishment  of  necessary  plans  and  resources  for  meeting 
objectives 


Have  decisions  about  the  use  of  one-shot,  incremental,  or  evolutionary 
development  been  validated  for  appropriateness  and  feasibility,  and 
accepted  by  the  key  stakeholders? 


Have  system  definition,  development,  test,  and  evolution  plans,  budgets, 
and  schedules  been  validated  for  appropriateness  and  feasibility,  and 
accepted  by  the  key  stakeholders? 

Is  there  a  valid  business  case  for  the  system,  relating  the  life  cycle  system 
benefits  to  the  system  total  cost  of  ownership? 


Project  designed  to  be  single  increment  FFP  contract. 

When  development  team  brought  on  and  began  to  validate 
initial  plans,  realized  that  the  project  was  not  do-able  and 
investigated  alternatives,  resulting  in  an  Ada  waiver,  the  use 
of  a  GUI  builder,  and  a  small  "agile"  development  team 
(that  also  produced  2167A  deliverables). 

Not  clear  what  better  evidence  would  be.  Initial  plans 
adjusted  and  presented  to  key  stakeholders  at 
requirements  review,  PDR,  and  CDR,  with  prototype  demo 
presented  by  PDR.  COCOMO  cost  model  not  useful  due  to 
nature  of  development  using  GUI  builder  (pre-COCOMO  II) 
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Critical  Success  Factor  2.4 
2.4(a) 

2.4(b) 


2.4(c) 


2.4(d) 


2.4(e) 


2.4(f) 


Establishment  of  appropriate  source  selection,  contracting,  and 
incentive  structures 

Has  the  competitive  prototyping  option  been  addressed,  and  the  decision 
accepted  by  key  stakeholders? 

If  doing  competitive  prototyping,  have  adequate  plans  and  preparations 
been  made  for  exercising  and  evaluating  the  prototypes,  and  for 
sustaining  core  competitive  teams  during  evaluation  and  down  selection? 


Is  the  status  of  the  contractor's  business  and  team  "healthy,"  both  in 
terms  of  business  indicators,  and  within  the  industrial  base  for  the 
program  area?  Is  the  program  aligned  with  the  core  business  of  the  unit, 
and  staffed  adequately  and  appropriately? 


Has  the  acquiring  organization  successfully  completed  projects  similar  to 
this  one  in  the  past? 

Has  the  candidate  performing  organization  successfully  completed 
projects  similar  to  this  one  in  the  past? 

Is  the  program  governance  process,  and  in  particular  the  system 
engineering  plan,  well  articulated  and  compatible  with  the  goals  of  the 
program? 


N/A--system  developed  using  known  technologies  and 
products 


Contractor's  established  team  at  company  headquarters 
started  the  development,  then  transitioned  development  to 
the  customer's  location  after  hiring  new  staff  for  the  local 
office.  New  hires  were  well-vetted  and  intially  worked  with 
headquarter's  team  through  a  transition  period.  Not  clear 
what  other  evidence  one  would  look  for. 

This  project  did  not  need  to  employ  "rocket  science".  It 
was  a  fairly  typically  IT  system  with  some  key,  but  well- 
understood,  external  interfaces. 

This  project  did  not  need  to  employ  "rocket  science".  It 
was  a  fairly  typically  IT  system  with  some  key,  but  well- 
understood,  external  interfaces. 

No  SEP  was  required  for  this  progam  (well  below  the  MDAP 
threshold) 


Critical  Success  Factor  2.5 


Establishment  of  necessary  personnel  competencies 

Does  the  government  have  access  over  the  life  of  the  program  to  the 
talent  required  to  manage  the  program?  Does  it  have  a  strategy  over  the 
life  of  the  program  for  using  the  best  people  available  in  the  government, 
the  FFRDCs,  and  the  professional  service  industry? 


Yellow  risk  button  not  enabled  here.  In  response  to 
questions,  sponsor  PM  was  a  computer  scientist  with  IT 
experience. 


At  Milestone  B,  have  sufficiently  talented  and  experienced  program  and 
systems  engineering  managers  been  identified?  Have  they  been 
empowered  to  tailor  processes  and  to  enforce  development  stability  from 
Milestone  B  through  IOC? 


Has  the  government  attempted  to  align  the  duration  of  the  program 
manager's  assignment  with  key  deliverables  and  milestones  in  the 
program? 

Is  the  quantity  of  systems  engineering  personnel  assigned,  their  skill  and 
seniority  mix,  and  the  time  phasing  of  their  application  throughout  the 
program  life  cycle  appropriate? 


Potentially  high  impact  due  to  the  FFP  nature  of  the 
contract.  Not  sure  what  "good"  or  "externally  validated" 
evidence  is  here.  People  selected  in  the  headquarter's 
office  were  "known  quantities"  within  the  development 
organization  and  staffed  the  project  with  a  new  team  at  the 
customer's  location.  New  staff  were  interviewed  and 
references  checked.  However,  I  would  not  call  this 
"externally  validated"  since  it  is  known  that  references  are 
hesitant  to  say  anything  negative  due  to  the  fear  of 
lawsuits. 

Not  a  major  issue  since  the  program  was  only  scheduled  for 
about  18  months. 

Not  really  a  major  issue  for  an  IT  application  using  COTS 
products. 


Goal  3: 


Technology  maturing,  architecting 


Critical  Success  Factor  3.1 


3.1(a) 


3.1(b) 


3.1(c) 


COTS/NDI/Services  evaluation,  selection,  validation  for  maturity  & 
compatibility 

Have  COTS/NDI/Services  opportunities  been  evaluated  prior  to  baselining 
requirements? 


Have  COTS/NDI/Services  scalability,  compatibility,  quality  of  service,  and 
life  cycle  support  risks  been  thoroughly  addressed? 


Has  a  COTS/NDI/Services  life  cycle  refresh  strategy  been  developed  and 
validated? 


Proposed  COTS  were  established  products 

At  the  time  the  products  were  selected,  they  were  probably 
adequately  evaluated.  However,  a  few  years  later,  ORACLE 
drove  several  other  DBMS  products  out  of  the  market 
place.  Not  clear  that  could  have  been  anticipated  at  the 
time  since  ORACLE  was  going  through  some  growing  pains 
at  the  time  this  project  was  initiated.  In  addition,  options 
were  limited  due  to  the  "color  of  money"  and  ORACLE  was 
not  thought  to  be  a  candidate  since  the  legacy  system  was 
not  built  upon  ORACLE. 

Seemed  reasonable  at  the  time  since  an  established  vendor 
was  used. 


Critical  Success  Factor  3.2 


Life-cycle  architecture  definition  &  validation 

Has  the  system  been  partitioned  to  define  segments  that  can  be 
independently  developed  and  tested  to  the  greatest  degree  possible? 

By  Milestone  A,  is  there  a  plan  to  have  internal  and  external  information 
exchange  protocols  established  for  the  whole  system  and  its  segments  by 
Milestone  B? 

Does  the  project  have  adequate  processes  in  place  to  define  the 
verification,  test  &  validation,  and  acceptance  of  systems  and  system 
elements  at  all  phases  of  definition  and  development? 

Is  there  a  clear,  consistent,  and  traceable  relationship  between  system 
requirements  and  architectural  elements?  Have  potential  off-nominal 
architecture-breakers  been  addressed? 

Does  the  architecture  adequately  reconcile  functional  hardware  "part-of" 
hierarchies  with  layered  software  "served-by"  hierarchies? 

Has  a  Work  Breakdown  Structure  (WBS)  been  developed  with  the  active 
participation  of  all  relevant  stakeholders,  which  accurately  reflects  both 
the  hardware  and  the  software  product  structure? 


Basis  for  work  assignments  across  the  programming  team. 

Used  existing  protocols  implemented  in  legacy  system 

The  "evidence"  seemed  reasonable  since  subject  matter 
experts  from  a  subcontractor  organization  were  used  and 
the  prime  supported  the  development  of  the  test  plan  and 
procedure  documentation. 

Not  really  an  issue  with  this  IT  system 

Not  really  an  issue  with  this  IT  system 

Not  really  an  issue  with  this  IT  system-all  hardward  was 
standard  COTS 
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Use  of  prototypes,  exercises,  models,  and  simulations  to 
determine  technological  solution  maturity 


Will  risky  new  technology  mature  before  Milestone  B?  Is  there  a  risk 
mitigation  plan? 


Have  the  key  non-technical  risk  drivers  been  identified  and  covered  by  risk 
mitigation  plans? 

Is  there  a  sufficient  collection  of  models,  and  appropriate  simulation  and 
exercise  environments,  to  validate  the  selected  concept  and  the  CONOPS 
against  the  KPPs? 

Has  the  claimed  degree  of  reuse  been  validated? 


Contract  was  awarded  at  Milestone  B  with  plan  to  use  no 
new  technologies.  When  it  was  decided  to  use  a  new-on- 
the-market  GUI  builder  after  contract  award,  the  tool  was 
tested  during  the  development  of  the  user  l/F  prototype 
before  a  final  decision  was  made. 


Not  really  an  issue  with  this  IT  system.  The  user  l/F  was 
prototyped  and  evaluated  by  the  key  users  prior  to 
completion  of  PDR. 

No  reuse  was  initially  planned  (is  this  a  duplicate  question?) 


Critical  Success  Factor  3.4 


Validated  system  engineering,  development,  manufacturing, 
operations  &  maintenance  budgets  and  schedules 


Are  the  major  known  cost  and  schedule  drivers  and  risks  explicitly 
identified,  and  is  there  a  plan  to  track  and  reduce  uncertainty? 

Have  the  cost  confidence  levels  been  developed  and  accepted  by  the  key 
system  stakeholders? 

Is  there  a  top-to-bottom  plan  for  how  the  total  system  will  be  integrated 
and  tested?  Does  it  adequately  consider  integration  facilities  development 
and  earlier  integration  testing? 

If  time-boxing  or  time-determined  development  is  used  to  stabilize 
schedules,  have  features  been  prioritized  and  the  system  architected  for 
ease  of  adding  or  dropping  borderline  features? 

Are  there  strategies  and  plans  for  evolving  the  architecture  while 
stabilizing  development  and  providing  continuity  of  services? 


High  impact  due  to  FFP  nature  of  contract,  but  were 
identified  and  managed  well  early-on 

Total  cost  was  negotiated  prior  to  contract  award. 

Development  lab  provided  at  contractor's  facility  that 
supported  both  development  and  test.  This  was 
established  prior  to  development  contract  award. 

N/A  due  to  single  increment 

Not  really  an  issue  with  an  IT/DBMS-based  application. 
What  might  have  been  more  of  an  issue  was  the  underlying 
data  model,  but  most  data  elements,  tables,  user  forms 
were  well  defined  early-on. 
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Goal  4: 


Evidence-based  progress  monitoring  and  commitment  reviews 


2 

2 

2 

2 

2 


Critical  Success  Factor  4.1 


Monitoring  of  system  definition  &  development  progress  vs.  plans  2 


Are  the  levels  and  formality  of  plans,  metrics,  evaluation  criteria,  and 
associated  mechanisms  (IMP,  IMS,  WBS,  EVMS)  commensurate  with  the 
level  of  project  requirements  emergence  and  stability?  (Too  little  is  risky 
for  pre-specifiable  and  stable  requirements;  too  much  is  risky  for 
emergent  and  unstable  requirements.) 


Level  of  formality  may  have  been  excessive  for  project. 
However,  due  to  the  FFP  nature  of  the  contract,  the 
development  organization  used  corporate  standards  for 
risky  programs  to  monitor  this  program. 


Are  the  project's  staffing  plans  and  buildup  for  progress  monitoring 
adequate  with  respect  to  required  levels  of  expertise? 

Have  most  of  the  planned  project  personnel  billets  been  filled  with  staff 
possessing  at  least  the  required  qualification  level? 

Is  the  project  adequately  identifying  and  managing  its  risks? 

Have  the  processes  for  conducting  reviews  been  evaluated  for  feasibility, 
reasonableness,  completeness,  and  assurance  of  independence? 

Has  compliance  with  legal,  policy,  regulatory,  standards,  and  security 
requirements  been  clearly  demonstrated? 


Critical  Success  Factor  4.2 
2  4.2(a) 

2  4.2(b) 

1  4.2(c) 

2  4.2(d) 

2  4.2(e) 


Monitoring  of  feasibility  evidence  development  progress  vs.  plans  2 

Has  the  project  identified  the  highest  risk  areas  on  which  to  focus 
feasibility  analysis? 

Has  the  project  analyzed  alternative  methods  of  evaluating  feasibility 
(models,  simulations,  benchmarks,  prototypes,  reference  checking,  past 
performance,  etc.)  and  prepared  the  infrastructure  for  using  the  most  cost- 
effective  choices? 

Has  the  project  identified  a  full  set  of  representative  operational  scenarios 
across  which  to  evaluate  feasibility? 


GUI  environment 


See  above 


Has  the  project  prepared  milestone  plans  and  earned  value  targets  for 
measuring  progress  in  developing  feasibility  evidence? 


Is  the  project  successfully  monitoring  progress  and  applying  corrective 
action  where  necessary? 
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Critical  Success  Factor  4.3 


Monitoring,  assessment,  and  replanning  for  changes  in  needs, 
opportunities,  and  resources 

Does  the  project  have  an  effective  strategy  for  performing  triage  (accept, 
defer,  reject)  on  proposed  changes,  which  does  not  destabilize  ongoing 
development? 

Does  the  project  have  an  adequate  capability  for  performing  change 
impact  analysis  and  involving  appropriate  stakeholders  in  addressing  and 
prioritizing  changes? 

Is  the  project  adequately  verifying  and  validating  proposed  changes  for 
feasibility  and  cost-effectiveness? 


2 


Strategy  defined,  but  seldom  used  on  project 


Process  detined,  but  seldom  used  on  project 


Only  received  a  couple  of  change  requests  during  project 


Critical  Success  Factor  4.4 


Use  of  milestone  reviews  to  ensure  stakeholder  commitment  to 
proceed 

Are  milestone  review  dates  based  on  the  availability  of  feasibility 
evidence,  instead  of  the  availability  of  artifacts  or  the  occurrence  of 
planned  review  dates? 

Are  artifacts  and  evidence  of  feasibility  evaluated,  and  risky  shortfalls 
identified,  by  key  stakeholders  and  independent  experts,  prior  to  review 
events? 

Are  developer  responses  to  identified  risks  prepared  prior  to  review 
events? 

Do  reviews  achieve  risk-based  concurrence  of  key  stakeholders  on 

whether  to  proceed  into  the  next  phase?  (Proceed;  skip  a  phase;  revisit  Not  an  option  after  FFP  contract  issued 

the  current  phase;  terminate  or  rescope  the  project.) 
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APPENDIX  C:  SECRT  EXAMPLE  -  LOGISTICS  SUPPORT  SYSTEM 

PROJECT 
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2 
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Question 

# 

Goal  1: 

Critical 

1.1(a) 

1.1(b) 

1.1(c) 

1.1(d) 

1.1(e) 

Critical 

1.2(a) 

1.2(b) 

1.2(c) 

1.2(d) 

Critical 

1.3(a) 

1.3(b) 

1.3(c) 


Impact 


Competency/Risk 


Q.  01 

x  .S2  <u 
CU  *-  c 

o  .SP  5 


NOTE:  Impact  and  evidence/risk  ratings  should  be  done  independently.  The  impact 

rating  should  estimate  the  effect  a  failure  to  competently  address  the  specified  item 

might  have  on  the  program.  The  competency  rating  should  specify  the  observed, 

historical  experience  and  competency  of  the  systems  engineering  staff  on  past  Risk 

programs  with  respect  to  the  specified  risk  item.  Exposure 


Rationale/ 
Source  of  evidence 


Concurrent  definition  of  system  requirements  and  solutions 


Success  Factor  1.1 


Success  Factor  1.3 


Understanding  of  stakeholder  needs:  capabilities,  operational  concept,  key 
performance  parameters,  enterprise  fit  (legacy).  Ability  to  analyze  strengths  2 

and  shortfalls  in  current-system  operations  via: 

Participatory  workshops,  surveys,  focus  groups 

Operations  research  techniques:  operations  data  collection  and  analysis 
Mission  effectiveness  modeling  and  simulation 
Prototypes,  scenarios,  stories,  personas 

Ethnographic  techniques:  Interviews,  sampled  observations,  cognitive  task  analysis 


Concurrent  exploration  of  solution  opportunities;  analysis  of  alternatives 
(AoAs)  for  cost-effectiveness  and  risk  (measures  of  effectiveness).  Ability  to 
identify  and  assess  alternative  solution  opportunities  via  experimentation  and 
analysis  of: 

Alternative  work  procedures,  non-materiel  solutions 


Purchased  or  furnished  products  and  services 
Emerging  technology 
Competitive  prototyping 


Limited  by  color  of  money 

Limited  by  color  of  money 

Not  prior  to  contract  award.  Project  too 
small 


System  scoping  &  requirements  definition  (external  interfaces;  memoranda  of 
agreement).  Ability  to  establish  system  scope  and  requirements  via: 

Cost-schedule-effectiveness  assessment  of  needs  vs.  opportunities 

Organizational  responsibilities,  authorities,  and  accountabilities  (RAAs)  assessment 

Appropriate  degrees  of  requirements  completeness,  consistency,  testability,  and 
variability  due  to  emergence  considerations 
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Critical  Success  Factor  1.4 


Prioritization  of  requirements  &  allocation  to  increments.  Ability  to  prioritize 
requirements  and  schedule  them  into  increments  based  on  considerations  of: 

Stakeholder  priorities  and  returns  on  investment 

Capability  interdependencies  and  requirements  emergence  considerations 
Technology  maturity  and  implementation  feasibility  risks 


Incremental  delivery  not  an  option.  Project 
too  small. 


Goal  2: 


System  life-cycle  organization,  planning,  and  staffing 


Critical  Success  Factor  2.1 


Establishment  of  stakeholder  life-cycle  responsibilities,  authorities,  and 
accountabilities  (RAAs)  (for  system  definition  &  system  development).  Ability 
to  support  establishment  of  stakeholder  RAAs  via  conduct  of: 

Organizational  capability  analyses 

Stakeholder  negotiations 

Operational  exercise  analyses 


Well  established  in  sponsor  IT  group 


Critical  Success  Factor  2.2 


Establishment  of  integrated  product  team  (IPT)  RAAs,  cross-IPT  coordination 
needs.  Ability  to  establish  IPT  RAAs  and  cross-IPT  coordination  mechanisms 
via: 

Risk  identification,  analysis,  and  prioritization 
Organizational  RAAs  and  skills  availability  assessment 
Risk  interdependency  analysis 
Risk  resolution  cost-benefit  analysis 


Not  really  done  due  to  size  of  project,  small 
team  co-located,  with  considerable  inputs 
from  stakeholders. 


Critical  Success  Factor  2.3 


Establishment  of  necessary  resources  for  meeting  objectives.  Ability  to 
support  program  negotiation  of  objectives  vs.  resources  via: 


Cost-schedule-capability  tradeoff  analyses 

Use  of  requirements  priorities  and  interdependencies  to  support  negotiation  of 
increment  contents 

Development  of  strategies  to  adjust  increment  content  to  meet  delivery  schedules 

Analysis  of  project  change  traffic  and  rebaselining  of  future-increment  plans  and 
specifications 


Not  much  negotiation  after  award  of  FFP 
contract.  However,  some  negotions 
2  conducted  in  order  to  incorporate  new 

higher  priority  requirement  and  drop  lower 
priority  requirements  for  a  no-cost  change. 


No  future  increments  planned 
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Critical 

2.4(a) 

2.4(b) 

2.4(c) 

2.4(d) 

Critical 

2.5(a) 

2.5(b) 

2.5(c) 

2.5(d) 

2.5(e) 

Goal  3: 

Critical 

3.1(a) 

3.1(b) 

3.1(c) 

3.1(d) 

3.1(e) 


Success  Factor  2.4 


Establishment  of  appropriate  source  selection,  contracting,  and  incentive 
structures.  Ability  to  support  program  management  in  preparing  source 
selection  materials,  matching  contracting  and  incentive  structures  to  program 
objectives,  and  technical  monitoring  of  performance  vs.  program  objectives: 

Preparation  of  proposal  solicitation  materials  and  evaluation  capabilities  and 
procedures 

Evaluation  of  proposal  submissions  with  respect  to  criteria 

Technical  support  of  contract  negotiations 

Technical  support  of  contract  performance  monitoring 


Success  Factor  2.5 


Assurance  of  necessary  personnel  competencies.  Ability  to  support  program 
management  in  evaluating  proposed  staffing  plans  and  monitoring  staffing 
capabilities  vs.  plans  in  the  areas  of: 

Concurrent  definition  of  system  requirements  &  solutions 

System  life  -cycle  organization,  planning,  and  staffing 

Technology  maturing  and  architecting 

Evidence-based  progress  monitoring  &  commitment  reviews 

Professional  and  interpersonal  skills 


N/A  due  to  size  of  project.  Project  done 
under  an  "umbrella"  services  contract. 


Good  staff  on  project,  but  system 
requirements/solution  approach  already 
established  by  contract  award.  Probably 
done  in  large  part  by  the  government. 


Technology  maturing,  architecting 


Success  Factor  3.1 


COTS/NDI  evaluation,  selection,  validation  for  maturity  &  compatibility. 
Ability  to  evaluate  alternative  combinations  of  COTS,  NDI,  and  purchased 
services  for: 

Functional  capabilities  vs.  system  needs 

Levels  of  service:  performance,  resilience,  scalability,  usability,  tailorability,  etc. 
Mutual  compatibility  and  external  interoperability 
Supplier  maturity,  stability,  support,  and  responsiveness 
Acquisition  and  operational  costs 


Key  was  identification  of  GUI  builder  to 
2  eliminate  Ada  requirment  for  interactive 
database  application. 


Done  using  vendor  benchmarks.  Selection  of 
vendor  limited  due  to  color  of  money  for 
COTS/NDI  not  an  issue  as  long  as  network 
connectivity  allowed 


Not  clear  how  to  rate  this  given  above 
constraints 
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Critical 

3.2(a) 

3.2(b) 

3.2(c) 

Critical 

3.3(a) 

3.3(b) 

3.3(c) 

3.3(d) 

Critical 

3.4(a) 

3.4(b) 

3.4(c) 

3.4(d) 


Success  Factor  3.2 


Life-cycle  architecture  definition  &  validation.  Ability  to  define  and  evolve 
configurations  of  hardware  and  software  components  and  connectors  along 
with  human  operational  architectures,  and  to  validate  that  they  cost- 
effectively  support  program  objectives: 

Define  candidate  hardware/software/human-operational  architectures 

Evaluate  their  functional  capabilities,  levels  of  service,  interoperability,  and 
sustainability  vs.  system  needs 

Perform  tradeoff  analyses  among  functional  capabilities  and  levels  of  service 


Only  one  hardware  configuration  based  on 
COTS 


Success  Factor  3.3 


Done  with  respect  to  GUI  builder 

Assess  relative  costs,  schedules,  and  capabilities  of  various  combinations  of  evaluation 
methods 

Prepare  plans  for  enabling  and  performing  evaluations 


Use  of  prototypes,  exercises,  models,  and  simulations  to  determine 
technological  solution  maturity.  Ability  to  assess  the  relative  costs  and 
benefits  of  alternative  evaluation  methods,  and  apply  appropriate 
combinations  of  methods: 


Prepare  representative  nominal  and  off-nominal  scenarios,  workload  generators,  virtual 
component  surrogates,  and  testbeds  to  support  evaluations 


Not  reasonable  for  small  project 


Perform  evaluations,  analyze  results,  investigate  anomalies,  and  adjust  plans  as  Not  done  with  any  rigor  due  to  size  of 

appropriate  project/schedule 


Success  Factor  3.4 


Validated  system  engineering,  development,  manufacturing,  operations  & 
maintenance  budgets  and  schedules.  Ability  to: 

Assess  alternative  budget  and  schedule  estimation  methods  vs.  nature  of  system, 
degree  of  system  knowledge,  complementarity  of  estimates,  and  cost  vs.  accuracy  of 
performing  estimates 

Prepare  plans  for  gathering  inputs  and  performing  estimates 

Perform  selected  combinations  of  estimates  and  reconcile  their  differences 

Perform  tradeoff  analyses  among  functional  capabilities,  levels  of  service,  costs,  and 
,  r  Not  an  option 

schedules 


Manufacturing  n/a.  Operations  not  within 
scope  of  contract-government  to  assume 
operations  and  maintenance  at  end  of 
contract  using  existing  staff 
maintaining/operating  system  being  replace. 
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Evidence-based  progress  monitoring  and  commitment  reviews 


Goal  4: 


Critical  Success  Factor  4.1 


Critical  Success  Factor  4.2 


Critical  Success  Factor  4.3 


Critical  Success  Factor  4.4 


Monitoring  of  system  definition,  development,  &  test  progress  vs.  plans. 
Ability  to  plan,  monitor,  and  evaluate  technical  progress  vs.  plans: 

Prepare  test  &  evaluation  facilities  &  plans  and  define  data  to  be  provided  for  assessing 
technical  progress  vs.  project  plans 

Monitor  performers'  technical  progress  in  developing,  verifying  and  validating  their 
technical  solutions 

Identify  shortfalls  in  technical  progress  vs.  plans,  and  determine  their  root  causes 

Monitoring  of  feasibility  evidence  development  and  test  progress  vs.  plans. 
Ability  to: 

Evaluate  developers'  feasibility  evidence  assessment  and  test  plans  for  coverage,  cost- 
effectiveness,  and  realism  of  assumptions 

Monitor  developers'  progress  with  respect  to  plans,  identify  shortfalls  and  root  causes 
Evaluate  feasibility  evidence  produced,  identify  shortfalls  and  root  causes 

Monitoring,  assessment,  and  replanning  for  changes  in  needs,  opportunities, 
and  resources.  Ability  to: 

Assess  proposed  changes  in  program  objectives,  constraints,  plans,  and  resources 

Perform  triage  to  determine  which  should  be  handled  immediately,  deferred  to  future 
increments,  or  rejected 

Perform  tradeoff  analyses  to  support  renegotiation  of  current  and  future  increment 
plans  and  contents 

Validate  feasibility  and  cost-effectiveness  of  renegotiated  increment  plans  and  contents 
Monitor  effectiveness  of  configuration  and  version  management 


Identification  and  mitigation  planning  for  feasibility  evidence  shortfalls  and 
other  technical  risks.  Ability  to  recommend  corrective  actions  for  feasibility 
evidence  shortfalls  and  technical  risks: 

Identify  and  evaluate  alternative  courses  of  action  to  address  feasibility  evidence 
shortfalls,  technical  risks,  and  root  causes 

Recommend  appropriate  corrective  actions  to  obtain  best-possible  system  outcomes 
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Critical 

4.5(a) 

4.5(b) 

4.5(c) 

Goal  5: 

Critical 

5.1(a) 

5.1(b) 

5.1(c) 

5.1(d) 

Critical 

5.2(a) 

5.2(b) 

5.2(c) 

Critical 

5.3(a) 

5.3(b) 

5.3(c) 


Use  of  milestone  reviews  to  ensure  stakeholder  commitment  to  proceed. 
Ability  to: 

Prepare  plans,  schedules,  budgets,  scenarios,  tools,  and  facilities  for  evaluating 
developer  feasibility  evidence 

Identify  shortfalls  in  feasibility  evidence  as  program  risks 

Assess  developer  risk  management  plans  for  coverage  of  risks,  identify  shortfalls,  and 
recommend  corrective  actions 


Professional  and  interpersonal  skills 


Success  Factor  5.1 


Leadership.  Ability  to  plan,  staff,  organize,  teambuild,  control,  and  direct 
systems  engineering  teams. 

Prepare  top-level  plans,  schedules,  budgets,  and  deliverables  for  a  system  engineering 
team 

Evaluate  and  recruit  appropriate  staff  members  for  executing  plans 

Involve  staff  members  in  collaborative  development  of  team  shared  vision,  detailed 
plans,  and  organizational  roles;  adjust  top-level  plans  as  appropriate 
Monitor  progress  with  respect  to  plans,  identify  shortfalls,  provide  mentoring  and 
constructive  corrective  actions  to  address  shortfalls 


Success  Factor  5.2 


Collaboration.  Ability  to  work  with  others  to  negotiate,  plan,  execute,  and 
coordinate  complementary  tasks  for  achieving  program  objectives 

Develop  understanding  of  other  participants'  value  propositions,  and  use  knowledge  to 
negotiate  mutually  satisfactory  roles,  responsibilities,  and  modes  of  collaboration 

Establish  modes  of  pro-active  coordination  of  emerging  issues  with  other  team 
members  and  teams 

Provide  help  to  others  in  need  of  your  capabilities 

Communication.  Ability  to  perform  timely,  coherent,  and  concise  verbal  and 
written  communication 

Develop  understanding  of  other  participants'  knowledge  boundaries  and  terminology, 
and  adjust  your  terminology  to  facilitate  their  understanding  of  your  communications 

Provide  timely,  coherent,  and  concise  verbal  and  written  communication  within  your 
team  and  among  external  stakeholders 

Explore  new  low-tech  (wallboards)  and  high-tech  (wikis,  blogs,  videos)  modes  of 
effective  communications 
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Critical  Success  Factor  5.4 


Accountability.  Ability  to  deliver  on  promises  and  behave  ethically 


Commit  to  and  follow  through  on  promised  commitments;  provide  advance  warning  of 
potential  delays  and  shortfalls 

Respect  the  truth,  intellectual  property,  and  the  rights  and  concerns  of  others 


Critical  Success  Factor  5.5 


Adaptability  and  Learning.  Ability  to  cope  with  uncertainty  and  unexpected 
developments,  and  to  seek  help  and  fill  relevant  knowledge  gaps 

Be  prepared  to  cope  with  inevitable  uncertainty  and  unexpected  developments 

Identify  key  knowledge  and  skills  needed  for  your  project  and  career,  and  engage  in 
learning  activities  to  master  them 


General  comment  on  tool  overall:  Too 
detailed  and  too  subject  to  interpretation.  It 
depends  on  who  is  going  to  do  the  evaluation 
and  what  criteria  they  are  using.  It  also 
depends  upon  the  size  and  scope  of  the 
project.  For  example,  it  will  be  difficult  for 
someone  who  does  not  already  know  the 
staff  to  evaluate  accountability...  and  an 
organization  is  not  typically  going  to  divulge 
the  fact  that  they  don't  think  all  of  their  staff 
are  accountable/ethical. 
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APPENDIX  D:  BASIC  COVERAGE  MATRIX 
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SERC  EM  Task  Coverage  Matrix  VI. 0 

NRC 

Probability 
of  Success 

SE  Leading 
Indicators 

LIPSF 

(Stevens) 

Anchoring 
SW  Process 
(USC) 

PSSES 

(U.  of  Alabama) 

SSEE 

(CMU/SEI) 

Macro  Risk 
Model/Tool 

Concept  Dev 

1 

Atleast  2  alternatives  have  been  evaluated 

X 

X 

X 

X 

(w.r.t  NPR) 

(x) 

2 

Can  an  initial  capability  be  achieved  within  the 
time  that  the  key  program  leaders  are  expected 
to  remain  engaged  in  their  current  jobs 
(normally  less  than  5  years  or  so  after  Milestone 
B)?  If  this  is  not  possible  for  a  complex  major 
development  program,  can  critical  subsystems, 
or  at  least  a  key  subset  of  them,  be 
demonstrated  within  that  time  frame? 

X 

(X) 

X 

X 

(5  years  is  not 
explicitly 
stated) 

(x) 

(seems  to  be 
inferrable  from 
the  conclusions) 

(x) 

(implies  this) 

3 

Will  risky  new  technology  mature  before  B?  Is 
there  a  risk  mitigation  plan? 

X 

X 

X 

(X) 

X 

X 

4 

Have  external  interface  complexities  been 
identified  and  minimized?  Is  there  a  plan  to 
mitigate  their  risks? 

X 

X 

X 

X 

X 

X 

KPP  and  CONOPS 

5 

At  Milestone  A,  have  the  KPPs  been  identified  in 
clear,  comprehensive,  concise  terms  that  are 
understandable  to  the  users  of  the  system? 

X 

(X) 

X 

(X) 

X 

(strongly 

implied) 

(X) 

(implied) 

X 

X 

6 

At  Milestone  B,  are  the  major  system-level 
requirements  (including  all  KPPs)  defined 
sufficiently  to  provide  a  stable  basis  for  the 
development  through  IOC? 

X 

X 

(X) 

X 

X 

(x) 

(X) 

(There  is  no 
direct  reference 

to  this  but  is 
inferrable) 

X 

7 

Has  a  CONOPS  been  developed  showing  that  the 
system  can  be  operated  to  handle  the  expected 
throughput  and  meet  response  time 
requirements? 

X 

X 

(X) 

(X) 

X 

(x) 

(there  is  a  mention 
of  a  physical 
solution.  That's  the 

closest  in  this 
regard) 

X 

X 

Legend: 

x  =  covered  by  EM 

(x)  =  partially  covered  (unless  stated  otherwise) 
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NRC 

Probability 
of  Success 

SE  Leading 
Indicators 

LIPSF 

(Stevens) 

Anchoring 
SW  Process 
(USC) 

PSSES 

(U.  of  Alabama) 

SSEE 

(CMU/SEI) 

Macro  Risk 
Model/Tool 

Cost  and  Schedule  Scoping 

8 

Are  the  major  known  cost  and  schedule  drivers 
and  risks  explicitly  identified,  and  is  there  a  plan 
to  track  and  reduce  uncertainty? 

X 

X 

(X) 

X 

(X) 

(seems  to  imply) 

(x) 

(They  aren't 
identified  per 
se.  It's  only 
"questioned"  as 
such.) 

X 

9 

Has  the  cost  confidence  level  been  accepted  by 
the  stakeholders  for  the  program? 

X 

X 

X 

X 

(x) 

X 

(not  directly 
stated) 

(X) 

10 

Is  there  a  sufficient  collection  of  models  and  an 
appropriate  simulation  environment  to  validate 
the  selected  concept  and  the  CONOPS  against 
the  KPPs? 

X 

X 

X 

X 

(X) 

(X) 

(seems  to  be) 

11 

At  Milestone  B,  do  the  requirements  take  into 
account  likely  future  mission  growth  over  the 
program  life  cycle? 

X 

X 

(X) 

(X) 

(x) 

X 

Architecture  dev 

12 

Has  the  system  been  partitioned  to  define 
segments  that  can  be  independently  developed 
and  tested  to  the  greatest  degree  possible? 

X 

(x) 

(not  directly 
stated  as 
such) 

X 

X 

13 

By  Milestone  A,  is  there  a  plan  to  have 
information  exchange  protocols  established  for 
the  whole  system  and  its  segments  by  Milestone 
B? 

X 

(x) 

Seems  far  fetched 
though 

(X) 

X 

14 

At  Milestone  B,  has  the  government  structured 
the  program  plan  to  ensure  that  the  contractor 
addresses  the  decomposition  of  requirements  to 
hardware  and  software  elements  sufficiently 
early  in  the  development  program? 

X 

(X) 

(nothing  specific  to 
the  contractor 
though) 

X 

Risk  Assessment 

15 

Have  the  key  risk  drivers  (not  only  the 
technology  drivers)  been  identified? 

X 

X 

X 

(X) 

Indirect 

inkling 

X 

(x) 

(majorly  technical) 

X 

X 
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NRC 

Probability 
of  Success 

SE  Leading 
Indicators 

LIPSF 

(Stevens) 

Anchoring 

SW  Process 
(USC) 

PSSES 

(U.  of  Alabama) 

SSEE 

(CMU/SEI) 

Macro  Risk 
Model/Tool 

Program  Implementation  Strategy 

16 

Does  the  government  have  access  over  the  life 
of  the  program  to  the  talent  required  to  manage 
the  program?  Does  it  have  a  strategy  over  the 
life  of  the  program  for  using  the  best  people 
available  in  the  government,  the  FFRDCs,  and 
the  professional  service  industry? 

X 

(X) 

(X) 

X 

17 

At  Milestone  A,  is  there  a  plan  defining  how  the 
pre-Milestone  B  activity  will  be  done,  and  by 
whom? 

X 

(X) 

X 

X 

(X) 

18 

Is  there  a  top-level  plan  for  how  the  total  system 
will  be  integrated  and  tested? 

X 

X 

(X) 

X 

X 

X 

X 

19 

At  Milestone  B,  have  sufficiently  talented  and 
experienced  program  and  systems  engineering 
managers  been  identified?  Have  they  been 
empowered  to  tailor  processes  and  to  enforce 
requirements  stability  from  Milestone  B  through 
IOC? 

X 

(X) 

X 

X 

(x) 

(loosely  connected) 

(x) 

X 

20 

Has  the  government  attempted  to  align  the 
duration  of  the  program  manager's  assignment 
with  key  deliverables  and  milestones  in  the 
program? 

X 

Miscellaneous 

21 

Status  of  Contractor's  business  and  his  team? 
Corporate  (about  the  organization)  and  or 
program  indicators  ( team  set  up  and  issues 
faced  etc). 

X 

X 

22 

Program  fit  in  capability  vision 

X 

(x) 

(X) 

23 

Staffing  and  manning  status/planning 

X 

X 

X 

X 

24 

Process  Compliance  (or  rationale  of  choice) 

(X) 

(allows  for 

process 

tailoring) 

X 

X 

(x) 

(X) 

(X) 

25 

Review  of  review  process 

(x) 

(Sharing  best 
practices  with 
other  agencies) 

(x) 

Pretty  close 

X 

(x) 

X 

(X) 

26 

Reuse  claim  validation 

(X) 

X 

X 

27 

Level  of  Service  Validation 

X 

(X) 
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NRC 

Probability 
of  Success 

SE  Leading 
Indicators 

LIPSF 

(Stevens) 

Anchoring 

SW  Process 
(USC) 

PSSES 

(U.  of  Alabama) 

SSEE 

(CMU/SEI) 

Macro  Risk 
Model/Tool 

28 

COTS  and  third  party  solutions 

X 

X 

X 

X 

29 

Technical  success/progress  measurement 

(X) 

X 

X 

X 

X 

30 

Verification/Validation  and  Configuration 
management 

X 

(V&V  seems 
implicit) 

X 

(X) 

31 

What  phases  of  the  SDLC  would  be  included  in 
developing  the  system 

(X) 

(implied  in 
process 
tailoring) 

X 

32 

Using  Integrated  Project  Teams  (IPT) 

X 

33 

Correlation  with  success  on  previous  projects 

X 

34 

Frequency  of  change  in  requirements 

(X) 

(Indirect 
reference  in 
CM) 

(x) 

X 

35 

About  the  contract  (contract  change  orders 
received,  %age  subcontracted  to  suppliers, 
current/initial  contract  value  of  project) 

X 

X 

36 

Stakeholder  Identification 

X 

X 

X 

37 

Questions  about  the  product  -  Who  is  acquiring 
this  product,  end  users,  Ul,  environment  of 
use/deployment  etc. 

(X) 

X 

(X) 

38 

Compliance  with  policy/standards/security  etc 

X 

X 

39 

Requirements/Architecture  Trace 

(X) 

X 

X 

X 

X 

(X) 

40 

Funding  Stability 

X 

(X) 

41 

Key  reviews  slip  more  than  30  days 

X 

X 

42 

Program  governance  process,  System  Eng.  Plan 
well  articulated 

(X) 

X 

X 

X 

X 

43 

Process  Improvement 

X 

(X) 

(mentioned  as  a 
part  of  process 
tailoring  effort) 

(X) 

(implied  in 
process 
strategies) 

X 

44 

Work  Breakdown  structure 

(x) 

(mentions  about 
work  products) 

X 

(X) 

(mention  about 
incremental 
development  of 
work  products) 

45 

Earned  Value  Management 

X 

(x) 

(no  direct  mention) 

(x) 

(talks  about 
"adding  value" 
and  ROD 

X 
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Systems  Engineering  Effectiveness  Measures  F 

DAPS  Methodology  v2.0 

1  =  Mission  Capabilites;  2  =  Resources;  3  =  Management; 

4  =  Technical  Process;  5  =  Performance;  6  =  Special  Interest  Areas 

General  Comment:  To  be  truly  complementary  to  the  DAPS,  the  questions  need  to  be  put  in  perspective  of  the  program 
life  cycle.  Are  these  questions  focused  on  the  contractor/supplier  or  the  government  acquisition  team?  It  is  often  difficult 
to  map  because  if  it  unclear  what  is  appropriate  when.  Also  need  to  look  at  terminology.  Often,  the  terminology  is  too 
vague  to  understand  what  is  really  intended  by  the  question. 

SE  EM  Framework  Area 

Interpretation 

DAPS 

DAPS  Topic 

Comments / 

References* 

Section 

Covered 

Observations 

*Not  every  reference  is  recorded  here. 

4 

1  Concurrent  definition  of  system 

requirements  and  solutions 

5 

6 

1.1 

Understanding  of  stakeholder 
needs:  capabilities,  operational 
concept,  key  performance 
parameters,  enterprise  fit 
(legacy) 

1.1 

CONOPS 

7 

1. 

At  Milestone  A,  have  the  KPPs 

Understandable, 

1.3.2 

KPPs  and  KSAs 

KPPs  are  required  to  be 

"1.3.2.Q7:  After  reviewing  the  table  with  the  program's  KPPs,  including  Net-Ready  and  Force 

1. 

been  identified  in  clear. 

comprehensive 

"established  and 

Protection  KPPs,  can  the  PMO  personnel  explain  the  rationale  for  the  thresholds  and  objectives?" 

1 

comprehensive,  concise  terms 

requirements 

documented";  no 

that  are  understandable  to  the 

guidance  on 

users  of  the  system? 

understandability/quali 
ty  of  KPP;  however 

states--> 

8 

1. 

Has  a  CONOPS  been  developed 

feasible  workload 

1.1 

CONOPS 

DAPS  specifies 

"The  material  from  a  CONOPS  will  feed  into  many  elements  of  information  required  by  the 

1. 

showing  that  the  system  can  be 

demonstrated  at 

1.2 

Analysis  of 

"realistic"  scenarios, 

Department  of  Defense  (DoD),  such  as  the  JCIDS  process,  the  Test  and  Evaluation  (T&E)  process. 

2 

operated  to  handle  both  nominal 

CONOPS;  are 

1.3.1 

Alternatives 

not  necessarily 

and  the  Analysis  of  Alternatives  (AoA). 

and  off-nominal  workloads,  and 

scenarios 

Reasonableness, 

"nominal  and  off- 

1.2.1.Q1:  How  were  mission  tasks  (MTs),  measures  of  effectiveness  (MOEs),  and  measures  of 

to  meet  response  time 

quantified 

Stability,  and 

nominal  workloads." 

performance  (MOPs)  derived  from  relevant  guidance  on  requirements  or  capabilities  (e.g.. 

requirements? 

Testability 

These  terms  and 

Mission  Needs  Statement  (MNS),  Operational  Requirements  Document  (ORD)  (if  pertinent),  or 

"response  time"  are 

the  problem  statement  found  in  the  ICD?  [1.2.1. Cl] 

more  network- 

•Are  they  quantifiable?  [1.2.1. Cl] 

oriented. 

1.2.1.Q5:  Are  the  threats  and  scenarios  realistic  and  current?  [1.2.1. Cl] 

Quantification  is  given 

1.3.1.Q5:  How  does  the  ICD  describe  the  threats  and  the  operational  environment  in  which  the 

by  "measures  of 

capabilities  are  to  be  exercised? 

effectiveness." 

Were  the  threats  and  scenarios  validated  by  the  Defense  Intelligence  Agency  (DIA)?  [1.3.1. Cl] 
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SE  EM  Framework  Area 

Interpretation 

DAPS 

DAPS  Topic 

Comments/ 

References* 

Section 

Covered 

Observations 

*Not  every  reference  is  recorded  here. 

10 

I- 

Has  the  ability  of  the  system  to 

verification  of 

1.2.1 

Validity  and 

1.2.1.Q2:  Are  the  MOEs  stated  in  terms  of  military  utility  and  based  on  value  provided  to  the 

i. 

meet  mission  effectiveness  goals 

mission  goals 

Currency  (1.2 

warfighter? 

3 

been  verified  through  the  use  of 

1.3.1 

Analysis  of 

•Are  these  MOEs  used  to  identify  models,  simulations,  and  other  analysis  tools  required  to 

modeling  and  simulation? 

4.4.2 

Alternatives) 

execute  the  study?  [1.2.1. Cl] 

Reasonableness, 

1.3.1.C12:  Verification  of  all  KPPs,  MOEs,  measures  of  suitability  (MOSs),  and  Critical  Technical 

Stability,  and 

Parameters  (CTPs)  are  demonstrated  by  prototypes  or  engineering  development  models 

Testability 

operating  in  the  system's  intended  environment.  Results  are  documented  in  test  and  evaluation 

Modeling  and 

reports  described  and  documented  in  accordance  with  the  Test  and  Evaluation  Master  Plan 

Simulation  Tools 

(TEMP).  Deficiencies  have  been  documented  and  analyzed,  and  the  associated  risks  for  successful 
testing  are  manageable. 

4. 4.2. Cl:  The  program  has  a  documented  modeling  and  simulation  (M&S)  approach  for  design 
and  analysis,  which  covers  its  purpose  and  use.  All  assumptions  and  weaknesses  inherent  in  the 
program's  M&S  activities  are  made  apparent  to  decision  makers.  This  approach  is  cross- 
referenced  in  the  Test  and  Evaluation  Master  Plan  (TEMP)  and  Systems  Engineering  Plan  (SEP). 

11 

1. 

Have  the  success-critical 

stakeholders 

3.3.3 

Management 

DAPS  does  not  address 

3.3.3.C3:  The  PMO  is  organized  to  execute  the  SDD  phase.  Program  IPTs  or  equivalent  are  formed 

1. 

stakeholders  been  identified  and 

identifed  and 

Structure  and 

the  "negotiation"  of 

and  will  include  all  appropriate  program  stakeholders  to  support  SDD  (ideally  these  IPTs  are 

4 

their  roles  and  responsibilities 

roles  defined 

Communications 

roles  and 

jointly  formed  with  the  contractor  IPTs).  The  organization  includes  support  from  the  acquisition 

negotiated? 

responsibilities.  This 

organization  infrastructure,  agencies  like  DCMA,  OSD,  and  from  contracted  support  personnel,  as 

term  implies  that  the 

required.  The  roles  and  responsibilities  are  clearly  defined  and  consistent  with  achieving  program 

person  performing  the 

objectives. 

task  has  the  option  to 

3.3.3.C5:  The  contractor  development  team  is  organized  to  execute  the  SDD  phase.  Program  IPTs 

choose  not  to  perform 

or  equivalent  are  formed  and  include  representatives  from  all  appropriate  stakeholders,  including 

some  part  of  the  task. 

the  PMO.  The  team  includes  support  from  the  company  infrastructure,  subcontractors  and 

In  DAPS,  the  PMO  is  in 

contracted  support  personnel,  as  required.  Roles,  responsibilities,  and  lines  of  authority  are 

charge  and  identifies 
what  needs  to  be  done 

and  who  shall  address 

it. 

clearly  defined  and  consistent  with  achieving  program  objectives. 

12 

1. 

Have  questions  about  the  fit  of 

system  fit  in 

1.3 

Capabilities: 

"For  a  materiel  solution  to  a  capability  requirement,  fielding  an  operational  capability  starts  with 

1. 

the  system  into  the  stakeholders' 

context;  scenarios 

4.1.6 

Perspective 

sound  strategies  for  requirements,  acquisition,  test  and  evaluation  (T&E),  and  support  and 

5. 

context  --  acquirers,  end  users. 

cover  spectrum  of 

Sustainment  as  a 

sustainment.  To  be  viable,  these  strategies  will  be  developed  in  concert  and  require  early  and 

administrators,  interoperators. 

occurrences  once 

Design 

ongoing  collaboration  among  operators,  developers,  acquirers,  testers,  sustainers,  and 

maintainers,  etc.  --  been 

fielded 

Consideration 

operations  analysts.  No  one  strategy  can  stand  alone  and  still  be  viable  because  all  are 

adequately  explored? 

interdependent  and  require  the  integration  of  the  others  to  be  effective." 

"4.1. 6. Q4:  How  is  the  program  planning  to  include  inputs  from  warfighters,  users,  developers, 
acquirers,  technologists,  testers,  budgeters,  and  sustainers  during  capability  needs 
develooment?" 

13 

14 

1.2 

Concurrent  exploration  of 

1.2 

Analysis  of 

solution  opportunities;  analysis 
of  alternatives  (AoAs)  for  cost- 
effectiveness  and  risk  (measures 
of  effectiveness) 

Alternatives 
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SE  EM  Framework  Area 

Interpretation 

DAPS 

DAPS  Topic 

Comments / 

References* 

Section 

Covered 

Observations 

*Not  every  reference  is  recorded  here. 

16 

1. 

Have  at  least  two  alternative 

alternative 

1.2 

Analysis  of 

Inferred,  by  definition 

2. 

approaches  been  explored  and 

approaches 

Alternatives 

of  AoA 

1 

evaluated? 

17 

1. 

At  Milestone  B,  has  the 

allocation  of 

4.5.1 

Software 

Pre-milestone  B:  4.5.1.C8:  Externally  visible  properties  of  the  system,  manifested  in  software  and 

2. 

government  structured  the 

capabilities  to 

Development  Plan 

hardware,  have  resulted  in  requirements  and  architecture  artifacts  that  have  been  carried 

2 

program  plan  to  ensure  that  the 

physical 

forward  from  Milestone  A  and  resulted  in  plans  and  technical  data  that  are  driving  requirements 

contractor  addresses  the 

architecture 

refinement,  design,  and  test  development. 

allocation  of  capabilities  to 

4.5.1.Q9:  Walk  through  the  architecture  and  design  of  the  system  as  known  now,  and 

hardware,  software,  and  human 

demonstrate  alignment  between  program  office  and  contractor  views.  Focus  particularly  on 

elements  sufficiently  early  in  the 

requirements  development  and  traceability,  identifying  artifacts  and  processes  that  demonstrate 

development  program? 

ongoing  alignment  among  the  program  office  and  contractor,  as  requirements  evolve  from 
externally  visible  (architecture)  properties  to  internally  visible  (design)  properties.  [4.5.1.C8] 

18 

1. 

Has  the  claimed  degree  of  reuse 

Actual  reuse. 

3.3.4 

Management 

3.3.4.3.C8:  The  Government  Program  Office  should  initially  approve  the  program  metrics  and 

2. 

been  validated? 

reuse  validated 

Methods,  metrics. 

then  periodically,  e.g.,  monthly,  the  metrics  should  be  reported  and  reviewed.  These  metrics 

3 

4.5.1 

and  Techniques 

should  include  many,  if  not  all  of  the  following:  Development  status  S  curves;  Processor 

Software 

throughput  utilization;  Processor  memory  utilization;  Input/output  utilization;  Software 

Development  Plan 

Engineering  Staffing;  Software  Work  Packages  Summary;  Schedule  Performance  Index;  Cost 
performance  Index;  Problem/Deficiencies  /Discrepancies  Status;  Requirements  Stability;  Software 
Size;  Software  Reuse  Status  (planned  versus  'actuals');  Reliability  Growth  Curve;  Logistics 

Footprint  Reduction;  Planned  Operational  Effectiveness;  Product  Availability  Predictions;  O&S 

Cost  Projections;  Development  Test  entrance  criteria  and  status;  DAES  Reporting  (For  MDAPS); 
Milestone  B  and  C  entrance  criteria. 

4.5. 1Q8:  Does  the  Software  Development  Plan  provide  for  early  demonstrations  (prior  to  the 
Preliminary  Design  Review  (PDR))  of  software  reuse  candidates  on  system  simulations?  [4.5.1.C7 
Reuse  of  software,  from  existing  systems  or  prior  development 

efforts,  has  been  analyzed  for  complexity  and  suitability  to  meet  required  functionality,  in 
accordance  with  accepted  software  engineering  standards.] 

19 

lT 

Have  the  claimed  quality  of 

validated  system 

4.4.2 

Modeling  & 

DAPS  does  not  use  the 

4.4.2. C3:  The  program  uses  M&S  during  the  Concept  Refinement  and  Technology  Development 

2. 

service  guarantees  been 

performance 

5.1 

Simulation  Tools 

term  "quality  of 

phases  to: 

4 

validated? 

Performance 

service";  assume  it  is 

...Identify  and  assess  the  system's  performance  in  its  intended  operating  environment  -  both 

Effectiveness 

the  same  as  system 

physical  (mechanical  and  electromagnetic)  and  operational  (information  exchange,  threat,  etc.) 

performance 

environments 

parameters 

5.1.1.C6:  Sufficient  CTPs  are  identified  in  the  TEMP  and  measure  critical  system  characteristics 
that,  when  achieved,  allow  the  attainment  of  desired  operational  performance  capabilities.  With 
each  technical  parameter,  thresholds  are  identified  for  each  stage  of  development. 
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21 

1. 

Have  proposed  COTS  and  third- 

COTS  maturity. 

1.2.1 

Validity  and 

Does  not  address  COTS 

1.2.1.Q9:  Does  the  prioritized  list  resulting  from  the  AMA  address  technological  maturity. 

2. 

party  solutions  been  validated  for 

suitability 

5.4.1 

Currency 

"validation" 

technological  risk,  supportability,  and  the  affordability  of  each  approach  using  the  best  available 

5 

maturity,  compatibility. 

Assessed 

specifically,  but  does 

data  in  the  pre-ICD  process?  [1.2.1.C2] 

supportability,  suitability,  and 

Manufacturing 

require  a 

5.4.1.C12:  Planned  non-developmental  items  (NDI)  or  commercial-off-the-shelf  (COTS)  items  have 

effectiveness,  throughout  the 

determination  of 

been  determined  to  meet  program  system  performance  and  sustainment  requirements  through  a 

expected  system  lifetime? 

suitabilty  through  an 

defined  acceptance  process. 

acceptance  process. 

5.4.1. Q12:  What  are  the  NDI  or  COTS  items  being  used  in  the  TD? 

•  What  are  the  sources  of  these  items? 

•  How  have  these  items  been  determined  to  meet  intended  program  performance  requirements? 
[5.4.1.C12] 

5.4.1. C24:  Planned  NDI  and  COTS  items  have  been  determined  to  meet  program  system 
performance  and  sustainment  requirements  through  a  defined  acceptance  process. 

5.4.1. C43:  Planned  NDI  or  COTS  items  have  been  determined  to  meet  program  system 
performance  and  sustainment  requirements  through  a  defined  acceptance  process. 

22 

23 

1.3 

System  scoping  &  requirements 

3.3.6 

Management  of 

3.3.6.C11:  The  boundary  and  scope  of  the  SoS  is  understood  by  the  PM  and  system  engineers  and 

definition  (external  interfaces; 

Dependencies  and 

the  SoS  is  adaptable  to  boundary  and  scope  changes  over  time.  All  systems  included  in  the  SoS 

memoranda  of  agreement) 

External  Interfaces 

should  be  identified.  Interfaces  from  the  SoS  to  external  systems  should  be  defined  and  scoped. 

(FoS/SoS) 

Specific  stakeholders  of  the  SoS  and  its  systems  should  be  identified,  including  their  organization. 
Identification  of  the  users  for  each  system  is  key. 

24 

1. 

Have  external  interface 

External  interface 

3.3.1 

Program 

3.3.1.Q32:  How  does  the  program  ensure  that  all  key  strategies  and  top-level  plans  remain 

3. 

complexities  been  identified  and 

agreements 

3.3.6 

Plan/Schedule 

consistent  and  aligned  (i.e.,  coordinated)  with  the  IMP/IMS? 

1 

minimized  via  MoAs  or  their 

Management  of 

•Are  the  type  and  number  of  technical  reviews  correct  in  each  appropriate  plan? 

equivalent?  Is  there  a  plan  to 

Dependencies  and 

•  Does  the  IMS  capture  both  the  government  SEP  and  the  prime  contractor's  SEMP/SEP  activities. 

mitigate  their  risks? 

External  Interfaces 

events,  and  milestones? 

(FoS/SoS) 

•  Are  the  scheduled  interfaces  w  FoS/SoS  correctly  captured  in  the  IMS,  SEP,  TEMP,  and  other 
related  plans? 

•  Did  the  plans  adequately  address  or  reference  all  key  processes  (e.g.,  Requirements,  Risk 
Management,  V&V,  Monitoring  &  Control,  Continuous  process  improvement,  etc.)?  [3.3.1.C5] 

3.3.6. Q5:  Does  the  SEP  and  TES  address  the  interface  interdependency  plans  for  development 
and  test.  [3.3.6.C6] 

3.3.6. Q14:  How  will  FoS/SoS  interfaces  be  managed?  And  what  is  the  plan  to  resolve  issues  that 
cross  PM,  PEO,  and  Service  lines? 

•  Have  Interface  Control  Documents  been  identified/developed  and  Interface  Control  Working 
Groups  been  assigned? 

•  Provide  a  summary  of  the  Memorandums  of  Agreement  (MOAs) 

•  Do  the  MOAs  include  any  "triggers"  that  require  a  FoS/SoS  member  to  inform  the 
others  if  there  is  a  cost,  schedule,  or  performance  deviation? 

3.3.6. C11:  The  boundary  and  scope  of  the  SoS  is  understood  by  the  PM  and  system 
engineers  and  the  SoS  is  adaptable  to  boundary  and  scope  changes  over  time.  All 
systems  included  in  the  SoS  should  be  identified.  Interfaces  from  the  SoS  to  external 
systems  should  be  defined  and  scoped.  Specific  stakeholders  of  the  SoS  and  its 
systems  should  be  identified,  including  their  organization.  Identification  of  the  users  for 
each  system  is  key. 
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26 

I- 

At  Milestone  B,  are  the  major 

requirement 

4.2.1 

Analysis  and 

4.2.1.C7:  System  requirements  specifications  and  performance  test/verification  requirements  are 

3. 

system-level  requirements 

definition  level 

Decomposition  (Pre 

linked  and  verification  methods  are  defined.  Note:  Allocation  of  system  functions  defines  the 

2 

(including  all  KPPs)  defined 

sufficient  for 

Milestone  B 

functional  baseline  of  the  system  design. 

sufficiently  to  provide  a  stable 

development 

criteria) 

•  Traceability  to  current  requirements  documentation  is  configuration  managed  for  approved 

basis  for  the  development 

capability  upgrades  commensurate  with  maturity  of  the  technology  required  for  the  upgrade. 

through  IOC? 

Maturity  is  verified  through  readiness  assessments  and  well-defined  metrics. 

•The  system  architecture  is  well  defined  and  documented,  and  is  in  accordance  with  all 
applicable  standards,  protocols  and  data  interchange  definitions  as  defined  by  key  interface 
descriptions. 

•Test  verification  descriptions,  critical  to  the  process,  are  defined  for  each  performance 
requirement. 

•Specifications  are  allocated  and  defined  to  the  appropriate  level  consistent  with  the  System 
Development  and  Demonstration  (SDD)  phase  objectives. 

4.2.1.C12:  -Software  requirements  are  evaluated  to  ensure  that  they  are  complete, 
unambiguous,  correct,  consistent,  verifiable,  modifiable,  traceable,  ranked  for  importance,  and 
ranked  for  stability.  Note:  Compliance  with  IEEE  Recommended  Practice  for  Software 

27 

1. 

By  Milestone  A,  is  there  a  plan  to 

Communication 

3.3.3 

Management 

3.3.3.C2:  The  PMO  organization  is  structured  to  interface  closely  and  openly  with  the  contractor 

3. 

have  information  exchange 

protocols 

Structure  and 

as  well  as  other  stakeholder  organizations.  The  PMO  leverages  other  government  organizations 

3 

protocols  established  for  the 

Communications 

to  benefit  the  TD  effort. 

whole  system  and  its  segments  by 

(pre-Milestone  A 

3.3.3.C6:  The  contractor  program  office  communicates  programmatic  information  internally  and 

Milestone  B? 

criteria) 

externally  in  a  timely  and  accurate  manner  across  the  contract  team  including  subcontractors. 

Management 

For  large,  geographically  distributed  system  development,  electronic  database  tools  are  used  to 

Structure  and 

support  this  communication.  The  participating  groups  and  functions,  including  production  and 

Communications 
(pre-Milestone  B 
criteria) 

support  functions,  are  tied  into  the  communication  channels  and  process. 

28 

1. 

Have  the  key  stakeholders  agreed 

Stakeholder 

4.2.2 

Management  of 

DAPS  seems  to  give  the 

4.2.2.Q8:  Do  the  stakeholders  understand  and  accept  all  the  requirements?  [4.2.2.C3] 

3. 

on  the  system  boundary  and 

agreement  on 

3.2.2 

Requirements 

mechanisms  to  do  so. 

3.2.2.C5:  Exit/Success  Criteria  from  CR  phase  -  all  reviews,  technical  and  programmatic  (i.e.,  ITR 

4 

assumptions  about  its 

requirements 

3.3.6 

Entrance  and 

but  does  not  always 

and  ASR),  in  support  of  specific  decision  points  have  been  successfully  conducted  with  valid 

environment? 

4.2.4' 

Exit/Success 

ask  if  agreement  has 

documentation,  data  and  analyses. 

Criteria 

been  achieved;  it 

3.2.2.Q21:  As  a  result  of  the  ASR,  does  the  resulting  set  of  requirements  agree  with  the  customer 

Management  of 

seems  to  be  assumed 

needs  and  expectations,  and  can  the  system  under  review  proceed  into  the  TD  phase?  [3.2.2.C5] 

Dependencies  and 

by  the  collaborative 

3.3.6.C12:  In  a  SoS  program,  the  technical  planning  process  must  be  initiated  top-down  but 

External  Interfaces 

efforts. 

iterated  within  individual  systems  until  a  consensus  approach  is  agreed  upon  and  resourced. 

Trade  Studies  and 

Systems  engineers  from  across  the  SoS  must  share  data  and  plans  and  engage  as  part  of  a 

Approaches 

collaborative  team  for  the  SoS.  It  is  important  to  recognize  the  value  of  a  collaborative  SE  team 
and  value  of  integration  facilities,  which  promote  open  and  active  exchange  and  experimentation 
among  members  of  the  SoS  SE  team. 

4.2.4.C2:  The  trade  space  (i.e.,  the  set  of  program  and  system  parameters,  attributes,  and 

characteristics  required  to  satisfy  performance  standards) 

has  been  identified  in  general  terms  and  agreed  to  by  the  stakeholders  - 

the  program  manager  (PM)  and  the  capability  needs  approval  authority. 

29 
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31 

1.4 

Prioritization  of  requirements  & 
allocation  to  increments 

4.2.2 

Management  of 
Requirements 

DAPS  often  mentions 

an  incremental 
approach  but  really 
does  not  mention 
anything  on  how  to 
divide  the 
requirements  into 

increments  or  to 

prioritize  requirements. 

It  seems  to  assume 
that  all  requirements 
will  be  implemented  as 
planned. 

4.2.2. C5:  The  program's  systems  engineering  (SE)  process  during  the  Technology  Development 
(TD)  phase  is  disciplined  in  documenting  and  tracking  specifications  at  all  levels,  and  structured  to 
manage  changes.  Integral  to  this  process  is  configuration  management  (CM).  The  CM  plan  lays 
out  the  process  and  plans  to  ensure  that  designs  are  traceable  to  requirements,  that  change  is 
controlled  and  documented,  that  interfaces  are  defined  and  understood,  and  that  there  is 
consistency  between  the  product  and  its  supporting  documentation. 

4.2.2. Q9:  How  does  the  requirements  management  plan  address  the  validation  of  requirements? 

•  How  are  the  prioritized  evaluation  criteria  consistent  with  requirements,  and  the  operations 
and  sustainment  concepts?  [4.2.2.C3] 

32 

1. 

4. 

1 

Can  an  initial  capability  be 
achieved  within  the  time  that  the 
key  program  leaders  are  expected 
to  remain  engaged  in  their 
current  jobs  (normally  less  than  5 
years  or  so  after  Milestone  B)?  If 
this  is  not  possible  for  a  complex 
major  development  program,  can 
critical  subsystems,  or  at  least  a 
key  subset  of  them,  be 
demonstrated  within  that  time 

frame? 

schedule 

feasibility 

3.1.1 

Acquisition 

Strategy/Credibility 

DAPS  does  not  address 

the  timeframe  of 
personnel  assignments, 
just  feasibility  within 
the  program's  schedule 

3.1.1.Q1:  How  is  the  Acquisition  Strategy  realistic? 

•  How  are  the  program  objectives  attainable? 

•  What  is  the  strategic  approach  to  attaining  the  program  objectives? 

•  Can  this  strategic  approach  be  successfully  implemented  with  reasonable  certainty?  Note: 

There  is  no  simple  formula  for  ensuring  the  approach  is  realistic.  To  evaluate  it,  reviewers  must 
perform  a  detailed  study  of  the  threat,  assess  the  state-of-the-art  in  all  technology  areas,  review 
past  performance  on  similar  acquisitions  or  systems,  and  survey  industry  capability,  then  attain 
consensus  on  the  complete  analysis.  Studies  take  time  and  resources,  but  because  realism  is  such 
an  important  criterion  for  a  successful  strategy,  every  effort  should  be  made  to  support  this 
undertaking  in  critical  areas  [3. 1.1. Cl] 

33 

1. 

4. 

2 

At  Milestone  B,  do  the 
requirements  take  into  account 
likely  future  mission  growth  over 
the  program  life  cycle? 

4.2.1 

4.2.2 

Analysis  and 
Decomposition 
Management  of 
Requirements 

Requires  "flexibility" 
for  change 

4.2.1. Q21:  What  are  the  features  of  the  design  architecture  that  will  ensure  it  remains  robust  and 
adaptable  throughout  the  system  life  cycle?  [4.2.1.C7] 

4.2.2. C4:  The  evolutionary  Acquisition  Strategy  (AS)  utilizes  a  management  system  that 
continually  defines  the  requirements  and  development  activities  to  support  the  evolving  needs; 
adequately  addresses  the  various  concerns  of  users,  developers,  and  managers;  and  mitigates  the 
risks  associated  with  these  issues  are  mitigated.  The  basic  system  architecture  is  designed  to 
accommodate  change.  Techniques  such  as  open  systems  design,  functional  partitioning  and 
modular  design  have  been  addressed  by  the  PM  to  achieve  a  flexible  system  that  can  be  easily 
and  affordably  modified. 
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35 

T~ 

4. 

3 

Have  appropriate  early  evaluation 
phases,  such  as  competitive 
prototyping,  been  considered  or 
executed  for  high-risk/low- 
maturity  components  of  the 
system? 

high-risk 

mitigation 

2.1 

3.3.1 

4.1.1 

4.3.1 

4.3.3 

Program  Schedule 
Overview 

Program 
Plan/Schedule 
System  Assurance 
Technical  Review 
Planning  (Pre 
Milestone  B) 
Baseline  Stability 

Emphasis  is  on 
identifying  the  risks 
and  does  not  say 
specificially  how  to 
handle  the  risks. 
"Prototypes  are  used 
as  part  of  the  SDD 
process,  as  are  reviews, 
methods,  and  tools." 

Perspective:  Experienced  program  personnel  provide  data  regarding  critical  and  high-risk  efforts 
and  identify  as  realistically  as  possible  the  expected  schedule,  which  the  program  management 
office  then  compares  with  the  top-level  defense  program  schedule  template  to  determine  the 
actual  schedule  risk  and  to  identify  all  schedule  drivers.  With  this  approach,  the  probability  of 
overrunning  a  program  schedule  can  be  estimated  by  determining  how  much  risk  exists  and 
where  it  is  greatest.  This  approach  enables  program  managers  (PMs)  to  estimate  early  and 
continuously  in  the  program  the  possibility  of  a  significant  likelihood  of  overrunning  the  program 
schedule  by  determining  how  much  and  where  the  risk  to  successful  schedule  completion  is 
greatest. 

"Early  industry  involvement  is  essential  in  the  identification  of  the  critical  and  high-risk  efforts  in 
the  development  of  the  integrated  schedule.  Integrated  scheduling  describes  the  detailed  tasks 
that  support  the  significant  activities  identified  in  integrated  planning  and  timing  of  tasks." 
Identifies  highest  risk  path  and  ensures  PM  is  applying  resources  on  it. 

3.3.1. Q21:  How  are  programs  with  high  risk  shown  in  the  IMS  in  order  to  give  the  visibility 
to  manage  and  control  risk?  [3.3.1.C2a] 

4.1.1. C3:  Pending  the  next  version  of  DoDI  5000.2,  "3. 5. 2. 6.  A  list  of  known  or  probable 

Critical  Program  Information  (CPI)  and  potential  countermeasures  such  as  Anti-Tamper 
(AT)  in  the  preferred  system  concept  and  in  the  critical  technologies  and  competitive 
prototypes  to  inform  program  protection  (DoDD  5200.39,  Reference  (ai))  and  design 
integration  during  in  the  TD  phase." 

"The  use  of  competitive  prototyping  is  required  by  the  Office  of  the  Under  Secretary  of 

Defense  for  Acquisition,  Technology  and  Logistics  (OUSD(AT&L))  policy  through  the 

Technology  Development  phase  up  to  Milestone  B,  which  will  include  the  Preliminary 

Design  Review." 

4.3.1. C13:  The  SRR  is  typically  held  well  in  advance  of  Milestone  B  to  allow  time  for 
issue  resolution  and  proper  executive  level  concurrence  on  process  and  results. 

4.3.3.C3:  The  functional  baseline  should  be  established  at  the  System  Functional  Review 
(SFR)  during  the  Technology  Development  phase.  Competitive  prototypes  of  system  or 
subsystem  components  should  be  developed  and  tested  to  ensure  program  requirements 
are  achievable. 

36 

iT 

4. 

4 

Have  stakeholders  agreed  on 
prioritization  of  system  features 
and  their  allocation  to 
development  increments? 

Prioritization  of 

capabilities/ 

requirements 

4.5.1 

4.2.1 

Software 

Development  Plan 
(Requirements) 
Analysis  and 
Decomposition 

Ranking  of 
requirements  is  only 
mentioned  in  the 
Software  Development 
section. 

4.5.1. Q4:  Walk  through  the  architecture  of  the  system  as  known  now,  and  demonstrate 
alignment  between  program  office  and  contractor  views.  Focus  on  requirements  traceability, 
from  initial  specification  of  capabilities  to  high-level  requirements  and  preliminary  architecture. 
[4.5.1.C3] 

4.5.1. Q9:  Walk  through  the  architecture  and  design  of  the  system  as  known  now,  and 
demonstrate  alignment  between  program  office  and  contractor  views.  Focus  particularly  on 
requirements  development  and  traceability,  identifying  artifacts  and  processes  that  demonstrate 
ongoing  alignment  among  the  program  office  and  contractor,  as  requirements  evolve  from 
externally  visible  (architecture)  properties  to  internally  visible  (design)  properties.  [4.5.1.C8] 

4.5.1. C11:  Software  process  integration  has  facilitated  timely  and  efficient  program  integration. 
Information  flow  has  not  been  impeded,  and  risks  traceable  to  information  flow  have  been 
perceived  and  mitigated  in  a  timely  fashion.  There  has  been  agreement  on  software  metrics  and 
plans  between  the  program  office  and  contractor. 

4.2.1. C12:  ^Software  requirements  are  evaluated  to  ensure  that  they  are  complete, 
unambiguous,  correct,  consistent,  verifiable,  modifiable,  traceable,  ranked  for 
importance,  and  ranked  for  stability. 

37 
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39 

2  System  life-cycle  organization,  planning,  am 

3.3.1 

Program 

Plan/Schedule 

40 

41 

2.1 

Establishment  of  stakeholder  life 

2.2.2 

Continuity  and 

2.2.2.Q3:  How  are  the  total  life  cycle  support  requirements  and  responsibilities  addressed  in  the 

cycle  responsibilities,  authorities, 

3.2.2 

Stability 

Logistics  Support  Plan?  What  are  the  bases  for  the  estimates?  [2. 2. 2. Cl] 

and  accountabilities  (RAAs)  (for 

Knowledge-based 

3.2.2.Q30:  In  preparation  for  the  IBR,  were  the  following  documents  provided  by  the  contractor 

system  definition  &  system 

3.3.1 

Decisions  and 

to  the  government  for  review? 

development) 

3.3.3 

Milestones 

•  Work  Authorization  Documents  (WADs) 

(Entrance  and 

•  Responsibility  Assignment  Matrix  (RAM) 

Exit/Success 

3.3.1.Q33:  How  is  the  SEP  updated  and  used  by  the  Technical  Leads  and  PM  to  manage  the 

Criteria) 

technical  aspects/efforts  of  the  program? 

Program 

•Was  the  SEP  prepared  in  time  to  support  RFPs? 

Plan/Schedule 

•Was  the  SEP  updated  after  contract  award  to  document  the  major  events,  revisions,  slips  in  the 

Management 

schedule,  technology  immaturity,  etc.  that  have  occurred?  Note:  The  SEP  is  the  PM's  overarching 

Structure  and 

technical  management  tool  that  reflects  both  government  and  contractor  activities,  roles,  and 

Communications 

responsibilities.  It  is  a  living  dynamic  plan,  updated  as  necessary  [3.3.1.C5] 

3.3.3. C5:  The  contractor  development  team  is  organized  to  execute  the  SDD  phase.  Program  IPTs 
or  equivalent  are  formed  and  include  representatives  from  all  appropriate  stakeholders,  including 
the  PMO.  The  team  includes  support  from  the  company 

infrastructure,  subcontractors  and  contracted  support  personnel,  as  required.  Roles, 
responsibilities,  and  lines  of  authority  are  clearly  defined  and  consistent  with  achieving 
program  objectives. 

3.3.3. Q9:  How  are  the  various  IPTs  organized  on  the  program,  and  do  they  have 
responsibility,  experienced  staff,  and  authority  to  make  decisions? 

42 

2. 

Are  the  stakeholders  who  have 

Qualified  staff 

2.3.1 

Sufficiency  of 

DAPS  addresses  the 

2.3.1. Cl:  There  is  an  established  program/process  in  the  program  management  office  (PMO)  that 

1. 

been  identified  as  critical  to  the 

Numbers  and 

issue  of  quality  staff  in 

provides  the  right  number  and  mix  of  qualified  personnel  to  successfully  execute  the  Technology 

1 

success  of  the  project 

Qualifications 

general,  but  does  not 

Development  (TD)  phase.  There  is  sufficient  flexibility  in  the  program  to  address  program 

represented  by  highly  qualified 

differentiate  with 

shortfalls  through  the  use  of  Systems  Engineering  Technical  Assistance  (SETA)  contractor 

personnel  --  those  who  are 

critical  areas;  also  does 

personnel. 

collaborative,  representative. 

not  define  what 

2.3.1.C2:  The  contractor  has  an  established  program  that  provides  the  right  number  and  mix  of 

authorized,  committed,  and 

qualified  people  are. 

qualified  personnel  to  successfully  execute  the  TD  phase.  Key  contractor  management  and 

knowledgeable? 

technical  personnel,  including  the  program  manager,  chief  systems  engineer,  software  architect, 
and  functional  area  managers,  have  worked  successfully  on  projects  of  similar  complexity  and 
have  had  significant  work  experience  relevant  to  the  current  program  phase. 
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44 

2~ 

1. 

2 

At  Milestone  A,  are  there 
validated  plans,  budgets,  and 
schedules  defining  how  the  pre- 
Milestone  B  activity  will  be  done, 
and  by  whom? 

Validated  plans 

3.3.1 

Program 

Plan/Schedule 

How  is  a  plan 
"validated"? 

3.3.1.  Cl:  The  Integrated  Master  Plan  (IMP)  is  an  event-driven  plan  that  documents  the  significant 
accomplishments  necessary  to  complete  the  work  and  ties  each  accomplishment  to  a  key 
program  event  that  forms  the  foundation  of  the  Integrated  Master  schedule  (IMS).  Note:  The  IMP 
Events  are  not  tied  to  calendar  dates;  each  event  is  completed  when  its  supporting 
Accomplishments  are  completed  and  as  evidenced  by  the  Criteria  completion  supporting  each  of 
those  Accomplishments. 

•Was  the  SEP  updated  after  contract  award  to  document  the  major  events,  revisions,  slips  in  the 
schedule,  technology  immaturity,  etc.  that  have  occurred?  Note:  The  SEP  is  the  PM's  overarching 
technical  management  tool  that  reflects  both  government  and  contractor  activities,  roles,  and 
responsibilities.  It  is  a  living  dynamic  plan,  updated  as  necessary  [3.3.1.C5] 

3.3.1. C6:  During  program  planning,  the  government  created  a  top  level  program  schedule  or 
Roadmap  which  provides  a  capstone  program  summary  allowing  insight  into  the  government's 
program  planning  and  approval  process.  This  initial  Roadmap,  along  with  other  program 
documentation,  provides  the  basis  for  an  initial  set  of  expectations  for  the  program  with  the  warfi 
•Is  prepared  by  the  government  program  office  early  in  the  program  planning  phase  in 
conjunction  with  any  other  supporting  or  associated  government  program  offices 

•  Is  focused  on  and  conveys  the  "big  picture"  of  the  program  objectives,  capabilities 
evolution,  summary  schedule,  and  any  major  program  constraints 

•Supports  initial  and  subsequent  budget  submissions  and  provides  the  basis  for 
developing  a  sound  position  on  funding  cuts  or  increases  throughout  the  program  life 

•  Contains  key  events  and  shows  critical  schedule  interfaces  (e.g.,  IOC  and  FOC)  with 
all  supporting  programs  and  activities  (for  example,  other  Services,  DARPA,  and  other 
agencies)  and  their  supporting  contracts 

•  Is  reviewed  regularly  by  the  primary  program  team  and  supporting  program  teams  to 
assess  progress  toward  accomplishing  key  event  and  schedule  interfaces 

•  Helps  detect  disconnects  early,  and  thus  provide  sufficient  lead-time  and  a  planning 
tool  to  help  address  them 

•  Is  able  to  be  traced  to  the  major  events  of  the  proposal  and,  upon  contract  award, 
trace  to  the  IMP/IMS 

•Is  kept  current 

45 

2. 

1. 

3 

Has  the  government  attempted 
to  align  the  duration  of  the 
program  manager's  assignment 
with  key  deliverables  and 
milestones  in  the  program? 

Assignment 

duration 

DAPS  does  not  address 

the  timeframe  of 
personnel  assignments, 
just  feasibility  within 
the  program's  schedule 
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2~ 

Have  the  key  stakeholders  agreed 

Roles  and 

3.3.6 

Management  of 

3.3.6.C12:  In  a  SoS  program,  the  technical  planning  process  must  be  initiated  top-down  but 

1. 

to  the  proposed  assignments  of 

responsibilities 

Dependencies  and 

iterated  within  individual  systems  until  a  consensus  approach  is  agreed  upon  and  resourced. 

4 

system  roles,  responsibilities,  and 

3.4.2 

External  Interfaces 

3.4.2.Q9:  How  has  the  PM  addressed  intra-government  work  agreements,  i.e.,  formal 

authorities? 

4.1.6 

Subcontractor 

agreements,  project  orders,  or  work  requests,  in  which  one  government  activity  agrees  to 

4.3.1 

management 

perform  work  for  another,  creating  a  supplier/customer  relationship? 

Sustainment  as  a 

3.4.2.Q24:  How  have  teaming  agreements  been  documented,  defined,  and  communicated  among 

Design 

all  relevant  parties? 

Consideration 

•What  is  the  process  for  making  changes  to  agreements,  and  who  is  involved?  [3.4.2.Cla] 

Technical  Review 

4.1.6.C15:  The  PM  shall  work  with  the  users  to  document  performance  and  support  requirements 

Planning 

in  performance  agreements  specifying  objective  outcomes,  measures,  resource  commitments, 
and  stakeholder  responsibilities.  The  military  Services  shall  document  sustainment  procedures 
that  ensure  integrated  combat  support. 

4.3.1.C30:  The  IBR  establishes  a  mutual  understanding  of  the  Performance  Measurement  Baseline 
(PMB)  and  provides  for  an  agreement  on  a  plan  of  action  to  evaluate  risks  inherent  in  the  PMB  an< 

48 

49 

2.2 

Establishment  of  integrated 

3.1.1 

Acquisition 

•  Use  of  Integrated  Product  Teams.  When  properly  oriented  and  challenged,  the  multifunctional 

product  team  (IPT)  RAAs,  cross- 

3.3 

Strategy/Credibility 

members  of  the  IPT  become  committed  to  program  success,  thereby  reducing  parochial  or 

IPT  coordination  needs 

Program  and 

Project 

functional  imbalances  that  could  otherwise  lead  to  future  instability.  [3.1.1. Cl] 

Management 

Integrated  product  and  process  development  is  a  management  process  that  integrates  all 
activities  from  the  concept  of  a  new  defense  system  through  the  entire  life  cycle,  using 
multidisciplinary  teams,  called  Integrated  Product  Teams  (IPTs). 

"The  DoD  has  recognized  the  importance  of  integrated  product  teams  as  a  means  to  aid  the 
program  manager,  and  as  a  way  to  streamline  the  decision  process.  By  working  as  part  of  cross¬ 
functional  teams,  issues  can  be  identified  and  resolved  more  quickly,  and  stakeholder 
involvement  in  the  overall  success  of  the  program  can  be  maximized.  In  this  way  the  program 
manager  capitalizes  on  the  strengths  of  all  the  stakeholders  in  the  defense  acquisition  system." 

50 

2. 

Does  the  project  make  effective 

Use  of  IPTs 

3.3.3 

Management 

3. 3. 3. Cl:  The  PMO  is  organized  to  execute  all  functions  in  preparation  for  Milestone  A  review  and 

2. 

use  of  Integrated  Project  Teams 

Structure  and 

TD  activities,  including  the  plan  for  formation  of  appropriate  Integrated  Product  Teams  (IPTs)  or 

1 

(IPTs)  throughout  the  supplier 

Communications 

their  equivalents.  Roles  and  responsibilities  are  clearly  defined  and  consistent  with  achieving  the 

hierarchy? 

TD  objectives. 

51 

2. 

Are  the  IPTs  staffed  by  highly 

Qualified  staff 

2.3.1 

Sufficiency  of 

Addresses  the  issue  of 

2.3.1. Cl:  There  is  an  established  program/process  in  the  program  management  office  (PMO)  that 

2. 

qualified  personnel,  as  in  2.a(a)? 

Numbers  and 

quality  staff  in  general. 

provides  the  right  number  and  mix  of  qualified  personnel  to  successfully  execute  the  Technology 

2 

Qualifications 

but  does  not 

Development  (TD)  phase.  There  is  sufficient  flexibility  in  the  program  to  address  program 

differentiate  with 

shortfalls  through  the  use  of  Systems  Engineering  Technical  Assistance  (SETA)  contractor 

critical  areas  or  IPTs 

personnel. 

2.3.1.C2:  The  contractor  has  an  established  program  that  provides  the  right  number  and  mix  of 
qualified  personnel  to  successfully  execute  the  TD  phase.  Key  contractor  management  and 
technical  personnel,  including  the  program  manager,  chief  systems  engineer,  software  architect, 
and  functional  area  managers,  have  worked  successfully  on  projects  of  similar  complexity  and 
have  had  significant  work  experience  relevant  to  the  current  program  phase. 
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53 

2~ 

For  IPTs  addressing  strongly 

Strongly  coupled 

Introduction 

DAPS  has  an 

"Quick-look"  reviews  are  conducted  2  to  3  months  before  the  milestone,  using  the  same  form 

2. 

coupled  objectives,  are  there 

3.2.1 

Statutory  and 

Overarching  Integrated 

and  formats  as  a  full  assessment.  They  are  conducted  as  a  "for  record"  review  to  support  the 

3 

"super-IPTs"  for  resolving 

Regulatory 

Product  Team  but  it  is 

Defense  Acquisition  Board's  (DAB)  Integrated  Process/Product  Teams  (IPTs),  Overarching 

conflicts  among  the  objectives? 

1.1 

Compliance  and 

unclear  on  this  group's 

Integrated  Product  Teams  (OIPTs),  or  if  requested,  for  the  DAB." 

4.1.2 

Guidance 

purpose  or 

3.2.1.Q5:  What  are  the  Service-specific  regulatory  requirements  for  the  program? 

Concept  of 

participation  in  the 

•  Are  they  in  conflict  with  higher  level  (e.g.,  DoD)  regulations? 

Operations 

program  during  all  life 

•  What  are  the  impacts  of  any  conflict  to  the  program? 

Modular  Open 

cycles. 

•  How  have  these  conflicts  been  resolved?  If  not,  were  the  conflicts  addressed  at  the 

Systems  Approach 

DAPS  mentions 

OIPT"Developing  the  CONOPS  as  a  team  effort  helps  resolve  requirement  debates  and  facilitates 
completeness  of  requirements." 

planning  for  resolving 

4.1.2.C2:  To  realize  open  systems  benefits,  programs  need  to  continually  measure  their  progress 

issues  throughout,  but 

toward  achieving  MOSA-enabled  capabilities/objectives.  Percentage  of  key  interfaces  defined  by 

does  not  mention 

open  standards,  or  percentage  of  components/subsystems  modularized  (self-contained. 

"super  IPTs"  per  se. 
DAPS  also  recommends 
MOSA  as  an  approach 
to  resolve  capability 
coupling 

decoupled,  and  encapsulated)  are  examples  of  open  systems-related  metrics. 

54 

55 

2.3 

Establishment  of  necessary 

3.3 

Program  and 

resources  for  meeting  objectives 

Project 

Management 
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2.  Have  decisions  about  the  use  of  Incremental  3.3  Program  and 

3  one-shot,  incremental,  or  development  3.3.1  Project 

^  evolutionary  development  been  Management 

validated  for  appropriateness  and  Program 

feasibility,  and  accepted  by  the  Plan/Schedule 

key  stakeholders? 


Comments/  References* * 

Observations  *Not  every  reference  is  recorded  here. 

In  general,  DAPS  uses  "Program  management .  . .  represents  the  integration  of  a  complex  system  of  differing  but 
the  focus  of  working  related  functional  disciplines  . . .  that  must  work  together  to  achieve  program  goals." 

together.  This  3.3.1.C6:  During  program  planning,  the  government  created  a  top  level  program  schedule  or 

probably  implies  Roadmap  which  .  . . 

"acceptance"  but  is  not  •  Is  focused  on  and  conveys  the  "big  picture"  of  the  program  objectives,  capabilities  evolution, 
explicitly  stated.  summary  schedule,  and  any  major  program  constraints 

3.3.1. Q36:  How  does  the  Government  Roadmap  Schedule  capture  the  plan  for  executing  the 
evolutionary  acquisition  (EA)  strategy,  with  either  a  spiral  or  incremental  development  process? 

3. 1.1.  Cl:  The  program  manager  (PM)  is  developing  a  credible  Acquisition  Strategy  that  will 
provide  the  basis  for  meeting  program  objectives  and  therefore  will  be  an  aid  in  gaining  program 
acceptance  and  support.  The  credibility  of  the  Acquisition  Strategy  is  evaluated  on  five 
attributes: 

•  Realism 

•  Stability 

•  Resource  balance 

•  Flexibility 

•  Managed  risk 

3.1.1. Q22:  How  does  the  Acquisition  Strategy  identify  and  describe  the  approach  the  program 
will  use  to  achieve  full  capability:  an  evolutionary  approach  or  a  single-step 

approach? 

•What  is  the  rationale  for  choosing  the  approach? 

•  If  an  evolutionary  approach  is  being  used,  how  is  Block  I  (the  initial  deployment 
capability)  described;  how  will  it  be  funded,  developed,  tested,  produced,  and  supported; 
and  what  is  the  approach  to  treatment  of  subsequent  blocks? 
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References* * 

*Not  every  reference  is  recorded  here. 

Before  all  milestones:  3.1.1.C2:  The  Acquisition  Strategy  is  credible,  based  on  the  following  five 
attributes:  realism,  stability,  resource  balance,  flexibility,  and  risk  management.  The  Acquisition 
Strategy  provides  the  basis  for  meeting  program  objectives,  thereby  acting  as  an  aid  in  gaining 
program  acceptance  and  support. 

3.1.1. C3:  The  Acquisition  Strategy  documents  the  ground  rules  and  assumptions  under  which  the 
program  was  started  and  upon  which  future  decisions  will  be  gauged.  It  becomes  more  definitive 
over  the  execution  of  the  program  in  describing  the  relationships  of  the  following  essential 
elements: 

•  Requirements 
•Structure  and  Schedule 
•Acquisition  Approach 

•  Risk  Management 
•Program  Management 

-Philosophy/Approach 
-Program  Resources 

-Information  Sharing  and  DoD  Oversight 
-Integrated  Digital  Environment  (IDE) 

-Defense  Contract  Management  Agency  (DCMA)  Support 
-Government  Property  in  the  Possession  of  Contractors  (GPPC) 

-Streamlining/Innovative  Acquisition 
-Simulation-Based  Acquisition  (SBA) 

-Software-Intensive  Programs 

•  Design  Considerations 
-Technology  Transition 

•Support  Strategy 

•  Business  Strategy  Competition 

1.1.1.  Cl:  The  system's  mission  description  clearly  identifies  mission  need,  objectives,  and  general 
capabilities.  Included  is  a  suitable  description  of  the  operational  (including  threat)  and  logistical 
environments  envisioned  for  the  system.  Information  is  current. 

1.2.1.  Cl:  There  is  a  viable  Analysis  of  Alternatives  (AoA)  study  plan  that  defines  what  will  be 
accomplished  and  how  it  will  be  done.  Minimum  information  in  the  study  plan  will  include: 
background,  purpose,  scope,  acquisition  issues,  alternatives,  effectiveness  and  cost 
methodologies,  analytical  tools,  and  schedule  to  complete  the  AoA. 

1.2.1. C2:  The  Analysis  of  Materiel  Approaches  (AMA),  if  conducted,  provides  the  best  materiel 
approach  or  combination  of  approaches  to  provide  the  desired  capability  or  capabilities.  The 
AMA  determines  the  best  way(s)  to  use  materiel  approaches  to  provide  a  joint  capability.  Note: 
Generally,  the  AMA  will  not  consider  which  specific  "systems"  or  "system  components"  are  best. 
3.4.3.C2:  There  is  a  viable  Value  Engineering  system  plan,  within  the  goals  and  stipulations  of  the 
PMO's  VE  program,  to  effectively  guide  the  successful  development  of 

solutions  that  eliminate  or  modify  any  element  of  the  program  that  significantly 
contributes  to  the  overall  cost  without  adding  commensurate  value  to  overall 
system  performance  or  program  execution. 


SE  EM  Framework  Area 

Interpretation 

DAPS 

DAPS  Topic 

Comments / 

References* 

Section 

Covered 

Observations 

*Not  every  reference  is  recorded  here. 

63 

2.4 

Establishment  of  appropriate 

3.4 

Contracting 

Figure  3-3  Contracting  Management  Process 

source  selection,  contracting. 

3.1 

Acquisition 

3.1.1.Q60:  What  contract  type(s)  are  identified  in  the  Acquisition  Strategy? 

and  incentive  structures 

Strategy 

•  Explain  why  the  contract  types  are  suitable,  including  considerations  of  risk  assessment  and 

(Credibility) 

reasonable  risk  sharing  by  the  government  and  the  contractor(s). 

•  How  does  the  strategy  explain  the  planned  contract  incentive  structure,  and  how  will  the 
contract  provide  incentives  for  the  contractor(s)  to  provide  the  contracted  product  or  services  at 
or  below  the  established  cost  objectives? 

-If  more  than  one  incentive  is  planned  for  a  contract,  what  is  the  explanation  of  how  the 
incentives  complement  each  other  and  do  not  interfere  with  one  another?  [3.1.1.C3] 

64 

2. 

Has  the  competitive  prototyping 

competitive 

4.1.1 

System  Assurance 

Competitive 

4.1.1.C3:  Pending  the  next  version  of  DoDI  5000.2,  "3. 5. 2. 6.  A  list  of  known  or  probable  Critical 

4. 

option  been  addressed,  and  the 

prototyping 

4.3.1 

Technical  Review 

prototyping  is  required. 

Program  Information  (CPI)  and  potential  countermeasures  such  as  Anti-Tamper  (AT)  in  the 

1 

decision  accepted  by  key 

Planning 

but  it  is  unclear  what 

preferred  system  concept  and  in  the  critical  technologies  and  competitive  prototypes  to  inform 

stakeholders? 

needs  to  be  done  for  it. 

program  protection  (DoDD  5200.39,  Reference  (ai))  and  design  integration  during  in  the  TD 
phase/' 

Pre-Milestone  B  (New  DoDI  5000.02) 

The  use  of  competitive  prototyping  is  required  by  the  Office  of  the  Under  Secretary  of  Defense 
for  Acquisition,  Technology  and  Logistics  (OUSD(AT&L))  policy  through  the  Technology 
Development  phase  up  to  Milestone  B,  which  will  include  the  Preliminary  Design  Review. 
4.3.1.Q20:  How  did  competitive  prototyping  affect  the  SRR? 

How  did  prototype  technical  performance  results  help  to  mature  the  system  requirements? 
Provide  some  examples.  [4.3.1.C13] 
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2. 

If  doing  competitive  prototyping. 

competitive 

4.3.1 

Technical  Review 

How  to  plan  for/use 

Not  addressed  to  this  level;  however  it  addressed  in  many  areas. 

4. 

have  adequate  plans  and 

prototyping  plans 

Planning 

competitive 

4.3.1.C13:  The  SRR  is  typically  held  well  in  advance  of  Milestone  B  to  allow  time  for  issue 

2 

preparations  been  made  for 

prototyping  in  a 

resolution  and  proper  executive  level  concurrence  on  process  and  results.  Technical  performance 

exercising  and  evaluating  the 

contract  or  program 

results  from  competitive  prototyping  should  factor  into  the  trade  space  for  system  requirements. 

prototypes,  and  for  sustaining 

management 

4.3.1.C21:  The  TD  effort  should  mature  the  prototype  technologies  to  an  acceptable  level  of  risk 

core  competitive  teams  during 

perspective  is  not  really 

to  proceed  to  EMDD  and  assessment  by  the  SRR.  The  results  of  the  TD  effort  should  be  reflected 

evaluation  and  down  selection? 

addressed.  However, 
the  results  of 
competitive 
prototyping  are 
used/evaluated  at  the 
major  reviews. 

97 


SE  EM  Framework  Area 

Interpretation 

DAPS 

Section 

DAPS  Topic 
Covered 

Comments/ 

Observations 

References* 

*Not  every  reference  is  recorded  here. 

67 

2~ 

4. 

3 

Is  the  status  of  the  contractor's 
business  and  team  "healthy," 
both  in  terms  of  business 
indicators,  and  within  the 
industrial  base  for  the  program 
area?  Is  the  program  aligned  with 
the  core  business  of  the  unit,  and 
staffed  adequately  and 
appropriately? 

contractor 

qualifications 

3.4 

Contracting 

The  wording  of  this 
item  is  confusing.  It 
seems  to  combine 
many  issues  into  one. 
How  is  "healthy" 
defined/determined? 
What  does  "within  the 

industrial  base  for  the 
program  area"  mean? 
Does  this  mean 
financial  stability  of  the 
company,  ability  to 
keep  qualified  staff, 
and  whether  this 
program  is  totally 
different  from  the 
company's  usual 
business? 

DAPS  coveres  whether  a  contractor  is  qualified  in  many  areas,  for  example, 

3.4.1. Q16:  In  regard  to  sources,  to  what  extent  has  the  acquisition  team  accomplished  the 
following  contracting  functions: 

•Availability  of  qualified  sources? 

•  Determination  if  the  source  can  meet  the  need? 

•  For  commercial  sources,  review  of  acquisition  histories,  conduct  of  market  research,  and 
preparation  of  source  lists  of  identified  sources? 

•Verification  that  a  Qualified  Bidders  List,  Qualified  Manufacturers  List,  or  Qualified  Parts  List 
(QBL/QML/QPL)  applies  to  the  procurement? 

•  Determination  from  market  research  whether  unlisted  firms  or  products  may  be  able  to  meet 
the  minimum  functional  need?  [3.4.1. C2b]3.4.1.Q56: 

3.4.1. Q56:  To  what  extent  were  past  performance,  technical  and  non-price  factors  addressed 
applied  by  the  acquisition  team? 

•  How  was  the  latest  performance  information  in  the  Service's  contractor  performance 
assessment  reporting  system  used? 

3.4.2. Cla:  The  PMO  and  contractor  have  adequately  addressed  pre-award  activities  during  the 
preparation  of  the  solicitation. 

-  Analysis  of  the  Industrial  Base  -  PM  has  determined  the  capabilities  of  the  national  technology  a 

3.4.2. Q5:  What  are  the  results  of  the  PM's  analysis  of  product  and  technology  areas 
critical  to  meeting  program  needs? 

•  How  does  the  Acquisition  Strategy  identify  the  potential  industry  sources  to  supply  these 
needs? 

•  Does  the  prime  contractor  plan  to  provide  critical  product  and  technology  areas  internally, 
by  subcontractor,  or  through  exclusive  teaming? 

3.4.2. Q31:  What  is  the  status  of  subcontractor  and  supplier  management  planning? 

•  How  are  audits,  supplier  ratings,  metrics,  value  stream,  etc.,  addressed  in  the  planning? 
[3.4.2.Clc] 

•  3.4.2.Q32:  How  is  the  past  performance  of  the  prime  contractors'  management  of 
subcontracts?  What  is  the  prime  contractor's  technical  capability  to  manage  the 
planned  subcontractors?  [3.4.2.Clc] 
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~2~ 

4. 

4 

Has  the  acquiring  organization 
successfully  completed  projects 
similar  to  this  one  in 
the  past? 

Acquisitioner's 

experience 

2.3 

Introduction 

Staffing  Level 

Program  Support  Team 

Structure:  The  Program  Support  Team  PST  is  composed  of  a  team  leader  from  SSE/AS  and  core 
subject  matter  expert  members  from  OSD  staff  (AT&L,  CAIG,  DPAP,  Networks  and  Information 
Integration  (Nil),  and  Director,  Operational  Test  and  Evaluation  (DOT&E)).  Additional  subject 
matter  experts  may  be  recruited  from  the  Services,  DoD  agencies.  Federally  Funded  Research  and 
Development  Centers  (FFRDCs),  and  academia  based  on  specific  assessment  needs  matched  with 
individual  expertise. 

-"The  key  to  applying  the  assessment  process  successfully  is  to  select  a  highly  qualified, 
experienced  team  leader,  and  populate  the  team  with  experienced  senior  individuals. 

Collectively,  the  assessment  team  should  bring  expertise,  experience,  and  knowledge  in  all  areas 
that  the  assessment  will  address." 

2.3:  Perspective:  Staffing  is  key  to  the  ability  of  any  PMO  to  execute  its  responsibilities. 

Composed  of  civilian,  military,  matrix  support,  and  Systems  Engineering  Technical  Assistance 
(SETA)  (aka  onsite  support  contractors),  the  staff  is  professional,  agile,  and  motivated.  It 
consistently  makes  smart  business  decisions,  acts  in  an  ethical  manner,  and 
delivers  timely  and  affordable  capabilities  to  the  warfighter.  A  successful  staff  is  more  than 
luck;  it  is  having  the  "right  person"  in  a  position,  rather  than  simply  filling  a  position.  The 
program  manager  (PM)  facilitates  this  success  through  improved  recruitment,  selection, 
and  training. 

2.3.1.Q4:  What  is  the  experience  level  of  each  of  the  existing  or  planned  key  technical 
personnel? 

How  is  the  experience  of  technical  personnel  relevant  to  the  current  activity?  [2.3.1. Cl] 
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2. 

4. 

5 

Has  the  candidate  performing 
organization  successfully 
completed  projects  similar  to  this 
one  in  the  past? 

contractor 

qualifications/ 

experience 

2.3.1 

Sufficiency  of 
Numbers  and 

Qualifications 

2.3.1.C4:  The  contractor  has  an  established  program  that  provides  the  right  number  and  mix  of 
qualified  personnel  to  successfully  execute  the  SDD  phase.  Key  contractor  management  and 
technical  personnel,  including  the  program  manager,  chief  systems  engineer,  software  architect, 
and  functional  area  managers,  have  worked  successfully  on  projects  of  similar  complexity  and 
have  had  significant  work  experience  relevant  to  the  current  program  phase.  The  contractor's 
policy  and  actual  practice  on  workforce  assignments  reflect  a  commitment  to  a  stable  workforce 
throughout  the  SDD  chase. 
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2. 

4. 

6 

Is  the  program  governance 
process,  and  in  particular  the 
system  engineering  plan,  well 
articulated  and  compatible  with 
the  goals  of  the  program? 

3.3.1 

Program 

Plan/Schedule 

The  term  "Program 
governance  process"  is 
confusing-not  sure 
how  this  relates  to  the 

system  engineering 
plan. 

Systems  Engineering  Plan  (SEP)  -  The  IMP  and  IMS  supports  the  sound  technical  approach 
documented  in  the  SEP.  The  IMP  demonstrates  the  contractual  commitment  to  the  elements  of 
major  technical  reviews  and  their  entry  and  exit  (success)  criteria.  The  SEP  and  IMS  demonstrate 
that  Cost,  Schedule  and  Performance  are  inter-related  within  the  program.  Note:  Because  the 
basic  tasks  within  the  IMS  track  the  systematic  flow  of  the  engineering  process,  there  should  be  a 
relationship  between  the  SEP  and  the  IMS.  These  processes,  tools,  and  documents  should  be 
understood,  linked,  and  tailored  for  an  individual  program's  execution  needs  and  management 
reporting  requirements 

3.3.1.Q32:  How  does  the  program  ensure  that  all  key  strategies  and  top-level  plans  remain 
consistent  and  aligned  (i.e.,  coordinated)  with  the  IMP/IMS? 

•Are  the  type  and  number  of  technical  reviews  correct  in  each  appropriate  plan? 

•  Does  the  IMS  capture  both  the  government  SEP  and  the  prime  contractor's  SEMP/SEP  activities, 
events,  and  milestones? 

•  Are  the  scheduled  interfaces  w  FoS/SoS  correctly  captured  in  the  IMS,  SEP,  TEMP,  and  other 
related  plans? 

•  Did  the  plans  adequately  address  or  reference  all  key  processes  (e.g..  Requirements,  Risk 
Management,  V&V,  Monitoring  &  Control,  Continuous  process  improvement,  etc.)? 

[3.3.1.C5] 
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2.5 

Establishment  of  necessary 
personnel  competencies 

2.3.1 

2.2.1 

Sufficiency  of 
Numbers  and 

Qualifications 
Program  Funding 
and  Allocation 

Flave  the  appropriate  technical  and  programmatic  competencies  been  involved  in  the  CARD-like 
document  development,  and  have  the  proper  subject  matter  experts  been  involved  in  its  review? 
(CARD-Cost  Analysis  Requirements  Description 

75 

2. 

5. 

1 

Does  the  government  have  access 
over  the  life  of  the  program  to 
the  talent  required  to  manage 
the  program?  Does  it  have  a 
strategy  over  the  life  of  the 
program  for  using  the  best 
people  available  in  the 
government,  the  FFRDCs,  and  the 
professional  service  industry? 

qualified 

Government  staff 

2.3.1 

Sufficiency  of 
Numbers  and 

Qualifications 

2.3.1.  Cl:  There  is  an  established  program/process  in  the  program  management  office  (PMO)  that 
provides  the  right  number  and  mix  of  qualified  personnel  to  successfully  execute  the  Technology 
Development  (TD)  phase.  There  is  sufficient  flexibility  in  the  program  to  address  program 
shortfalls  through  the  use  of  Systems  Engineering  Technical  Assistance  (SETA)  contractor 
personnel. 

2.3.1. Q6:  Are  the  personnel  (e.g.,  program  management,  contracting,  oversight)  trained  to  the 
appropriate  levels  in  accordance  with  their  acquisition  career  assignments? 

•  Are  government  PMO  personnel  in  acquisition-critical  positions  trained  to  the  appropriate 
certification  levels  in  accordance  with  their  acquisition  career  assignments?  [2.3.1. Cl] 

76 

2. 

5. 

2 

At  Milestone  B,  have  sufficiently 
talented  and  experienced 
program  and  systems  engineering 
managers  been  identified?  Flave 
they  been  empowered  to  tailor 
processes  and  to  enforce 
development  stability  from 
Milestone  B  through  IOC? 

qualified  program 
management, 
system 
engineering; 
empowered  staff 

2.3.1 

3.3.6 

Sufficiency  of 
Numbers  and 
Qualifications  (pre¬ 
milestone  B) 
Management  of 
Dependencies  and 
External  Interfaces 

The  two  questions 
should  be  separated. 
Empowerment  is 
different  from  qualified 
persons. 

2.3.1. C3:  The  PMO  staff  is  the  right  mix  of  qualified  personnel  to  successfully  execute  the  System 
Development  and  Demonstration  (SDD)  phase.  Workforce  management  and  training  programs 
receive  the  highest  priority  in  resources  to  ensure  a  qualified  workforce  to  complete  the  SDD 
phase  and  transition  to  production.  There  is  sufficient  flexibility  in  the  program  to  address 
program  shortfalls  through  the  use  of  SETA  contractor  personnel.  Policies  and  standards  are  in 
place  to  ensure  the  thorough  and  continual  training  of  the  workforce  and  to  evaluate  worker 
performance. 

2.3.1. C4:  The  contractor  has  an  established  program  that  provides  the  right  number  and  mix  of 
qualified  personnel  to  successfully  execute  the  SDD  phase.  Key  contractor  management  and 
technical  personnel,  including  the  program  manager,  chief  systems  engineer,  software  architect, 
and  functional  area  managers,  have  worked  successfully  on  projects  of  similar  complexity  and 
have  had  significant  work  experience  relevant  to  the  current  program  phase.  The  contractor's 
policy  and  actual  practice  on  workforce  assignments  reflect  a  commitment  to  a  stable  workforce  t 
3.3.6.Q11:  •  Is  the  SE&I  lead  empowered  to  integrate  the  programs  within  the  SoS,  and 
reallocate  resources  (e.g.  funding  and  manpower)  within  the  SoS  from  the  "fast  movers" 

to  the  "slow  movers"  program  to  keep  the  establishment  of  the  SoS  capability  on  track? 

77 

2. 

5. 

3 

Has  the  government  attempted 
to  align  the  duration  of  the 
program  manager's  assignment 
with  key  deliverables  and 
milestones  in  the  program? 

Length  of  assignment  is 
not  addressed  in  DAPS 
other  than  indirectly  by 
"realism"  guidelines 

Length  of  assignment  is  not  addressed 

78 

2. 

5. 

4 

Is  the  quantity  of  systems 
engineering  personnel  assigned, 
their  skill  and  seniority  mix,  and 
the  time  phasing  of  their 
application  throughout  the 
program  life  cycle  appropriate? 

2.3.1 

Sufficiency  of 
Numbers  and 
Qualifications  (pre¬ 
milestone  B) 

2.3.1.Q14:  How  has  the  contractor  committed  to  having  a  quality  workforce  throughout  the  TD 
phase?  [2.3.1.C2] 

Also,  see  above. 
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81 

3  Technology  maturing,  architecting 

82 

83 

3.1 

COTS/NDI  evaluation,  selection. 

3.1.1 

Acquisition 

validation  for  maturity  & 
compatibility 

Strategy 

84 

3. 

Have  COTS/NDI/Services 

COTS/NDI 

3.1.1 

Acquisition 

3.1.1.Q52:  How  does  the  Acquisition  Strategy  consider  both  commercial  and  NDI  sources  as  the 

1. 

opportunities  been  evaluated 

evaluation 

Strategy 

primary  source  of  supply?  What  is  the  role  of  market  research  in  determining  the  availability  and 

1 

prior  to  baselining  requirements? 

(Credibility) 

suitability  of  commercial  and  NDIs,  and  to  what  extent  do  the  interfaces  for  these  items  have 
broad  market  acceptance,  standards-organization  support,  and  stability? 

•What  is  the  role  of  commercial  off-the-shelf  (COTS)  and  NDI  sources  of  supply  to  provide  for  the 
most  cost-effective  system  throughout  the  system's  life  cycle? 

•  How  does  the  PM  work  with  the  user  to  define  and  modify,  as  necessary,  requirements  to 
facilitate  the  use  of  COTS  items  and  NDIs?  [3.1.1.C31 

85 

3. 

Have  COTS/NDI/Services 

3.1.1 

Acquisition 

Suitability  is  addressed 

1. 

scalability,  compatibility,  quality 

Strategy 

which  should  include 

2 

of  service,  and  life  cycle  support 

(Credibility) 

these  items,  but  all  of 

risks  been  thoroughly  addressed? 

them  are  not  called  out 
separately. 
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3. 

Has  a  COTS/NDI/Services  life  cycle 

COTS  refresh 

3.1.2 

Acquisition 

3. 1.2. Cl:  Before  development  of  a  program  Acquisition  Strategy  in  the  Technology  Development 

1. 

refresh  strategy  been  developed 

Strategy 

(TD)  phase,  a  Technology  Development  Strategy  (TDS)  is  formulated  during  the  Concept 

3 

and  validated? 

(Acceptability) 

Refinement  (CR)  phase  and  approved  by  the  MDA  at  Milestone  A.  The  TDS  contains  the  research 
and  development  strategy  to  be  implemented— particularly  in  the  TD  phase— and  the  rationale 
for  the  planned  acquisition  approach  to  achieve  full  capability. 

3.1.2. Q12:  How  is  technology  obsolescence  factored  into  the  TDS? 

•  Does  the  strategy  include  a  process  to  determine  when  technology-refresh  actions  should  be 
performed?  If  not,  why  not?  [3. 1.2. Cl] 

3.1.2. Q29:  How  does  the  Acquisition  Strategy  describe  the  program's  approach  for  applying 
Modular  Open  Systems  Approach  (MOSA),  as  characterized  by  the  following  attributes? 

...•Commercial-off-the-shelf  (COTS)  technology  refreshment  plans 

87 
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3.2 

Life-cycle  architecture  definition 
&  validation 

4.1.3 

Architecture 
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3. 

Has  the  system  been  partitioned 

system 

4.2.1 

Analysis  and 

The  wording  of  this 

4.2.1.C4:  The  program  manager  (PM)  or  contractor  has  an  effective  systems  engineering  (SE) 

2. 

to  define  segments  that  can  be 

partitioning 

Decomposition 

question  is  more 

process  in  place  to  perform  functional  analysis  and  the  allocation  of  functional  requirements  for 

1 

independently  developed  and 

software-oriented;  the 

the  TD  phase.  This  includes  the  traceability  and  verification  of  requirements  across  the  entire 

tested  to  the  greatest  degree 

DAPS  is  more  system- 

system. 

possible? 

oriented 

•  The  SE  process  is  effective  in  defining  system  requirements,  functionality,  and  allocated  physical 
architecture. 

•Technology  maturity  requirements  are  appropriately  scoped  for  demonstration  during  the  TD 
phase. 

•Analyses  provide  a  clear,  detailed  description  of  the  technical  approach  resulting  from 
functional  analysis  and  allocation. 

•  The  SE  process  uses  rigorous  and  disciplined  definitions  of  interfaces,  and  defines  the  key 
interfaces  that  require  test  verification  within  the  system. 

•  The  SE  process  partitions  the  system  into  self-contained,  groupings  of  interchangeable  and 
adaptable  modules.  The  process  enables  identification  of  key  test  and  evaluation  (T&E) 
requirements  to  verify  sub-assembly  performance  during  the  TD  phase. 
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3. 

By  Milestone  A,  is  there  a  plan  to 

information 

1.3.2 

Key  Performance 

Pre  milestone  B:  1.3.2.C8:  Programs  will  use  standardized  architectural  products  and  conventions. 

2. 

have  internal  and  external 

exchange 

parameters  and 

data  formats,  and  open  interface  standards  and  protocols  to  enable  interoperability. 

2 

information  exchange  protocols 

4.2.1 

Key  System 

Pre  milestone  A:  4.2.1. Cl:*  The  system  architecture  is  well  defined  and  documented,  and  is  in 

established  for  the  whole  system 

Attributes 

accordance  with  all  applicable  standards,  protocols  and  data  interchange  definitions  as  defined 

and  its  segments  by  Milestone  B? 

Analysis  and 
Decomposition 

by  key  interface  descriptions. 
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3. 

Does  the  project  have  adequate 

4.6.1 

Test  and  Evaluation 

4.6.1. Cl:  The  program  manager  (PM)  shall  develop  a  robust  integrated  Test  and  Evaluation 

2. 

processes  in  place  to  define  the 

Plan 

Strategy  (TES)  for  all  phases  of  the  program,  describing  developmental  test  and  evaluation 

3 

verification,  test  &  validation,  and 

(DT&E),  operational  test  and  evaluation  (OT&E),  and  live-fire  test  and  evaluation  (LFT&E). 

acceptance  of  systems  and 

Without  compromising  rigor,  the  program  is  required  to  integrate  modeling  and  simulation 

system  elements  at  all  phases  of 

(M&S)  activities  into  the  strategy.  The  TES  should  be  consistent  with  and  complementary  to  the 

definition  and  development? 

Systems  Engineering  Plan  (SEP). 
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3. 

Is  there  a  clear,  consistent,  and 

1.3.1 

Capatilities- 

In  DAPS,  it  is  not 

1.3.1. CIO:  Compatibility  with  other  interfacing  systems  is  maintained  and  verified  through  system- 

2. 

traceable  relationship  between 

Reasonableness, 

explicitly  required  to 

level  testing  as  defined  in  interface  specifications,  through  the  development/design  process,  and 

4 

system  requirements  and 

4.1.3 

Stability,  and 

have  traceability 

is  traceable  to  the  architecture  of  the  system.  Interface  specifications  are  under  formal 

architectural  elements?  Have 

4.2.1 

Testability 

between  reqs  and 

configuration  control. 

potential  off-nominal 

Design 

architecture 

4.1. 3. Q3:  How  is  a  change  within  the  technical  system  descriptions  ensured  for  traceability  of 

architecture-breakers  been 

Considerations/Arc 

impact  across  the  system?  [4.1. 3. C3] 

addressed? 

hitecture 

4.2.1.C4:  The  program  manager  (PM)  or  contractor  has  an  effective  systems  engineering  (SE) 

Analysis  and 

process  in  place  to  perform  functional  analysis  and  the  allocation  of  functional  requirements  for 

Decomposition 

the  TD  phase.  This  includes  the  traceability  and  verification  of  requirements  across  the  entire 
system. 

•  The  SE  process  is  effective  in  defining  system  requirements,  functionality,  and  allocated  physical 
architecture. 

4.2.1. CIO:  The  PM  or  contractor  has  an  effective  SE  process  in  place  to  perform  functional 
analysis  and  the  allocation  of  functional  requirements  for  the  SDD  phase.  This  includes  the 
traceability  and  verification  of  requirements  across  the  entire  system. 

•The  SE  process  is  effective  in  defining  system  requirements,  functionality,  and  allocated 
physical  architecture. 
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3~ 

Does  the  architecture  adequately 

Architecture 

4.1.3 

Architecture 

Unclear  of  the 

DAPS  not  written  this  way--think  the  following  covers  it: 

2. 

reconcile  functional  hardware 

decomposition 

terminology  used  in 

4.1. 3. Cl:  The  system  architecture  and  subsystem  architecture,  including  computer  system  and 

5 

"part-of"  hierarchies  with  layered 

this  question 

support  architectures,  is  defined  using  standardized  methods,  such  as  the  Department  of  Defense 

software  "served-by"  hierarchies? 

Architecture  Framework  (DoDAF),  and  widely  accepted  tools  sets,  such  as  those  that  employ  the 
Unified  Modeling  Language  (UML),  which  meets  the  system  requirements,  including  open-system 
requirements  and  benefits.  Ease  of  change,  growth,  upgrade,  and  lifecycle  support  is  facilitated 
with  this  architecture. 

4.1. 3.  C2:  The  technical  system  architecture  descriptions  should  use  mandated  Operational  View 
(OV),  System  View  (SV),  and  Technical  View  (TV)  products  as  described  in  the  DoDAF,  and  should 
be  integral  to  the  system  design.  There  should  be  System  Description  Documents  (SDDs)  and 
System  Capability  Specifications  (SCSs)  that  address  those  for  the  system  and  major  subsystems. 

4.1. 3.  C3:  There  should  be  a  disciplined  process  to  ensure  that  the  technical  system  descriptions 
are  integrated  such  that  changes  to  any  one  that  affects  others  is  identified  and  tracked  to  conclu; 
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3. 

Has  a  Work  Breakdown  Structure 

WBS 

3.3.2 

Work  Breakdown 

3.3.2.Q2:  For  a  joint  and/or  System  of  Systems  (SoS)  program,  does  the  WBS  identify  and  describe 

3. 

(WBS)  been  developed  with  the 

Structure 

the  "parent-child"  type  relationship?  Note:  Understanding  the  parent-child  type  relationship  of 

6 

active  participation  of  all  relevant 

various  related  programs  and  contracts  and  their  impact  on  the  WBS  is  important  in  the  ever- 

stakeholders,  which  accurately 

increasing  integrated  and  joint  program  environment.  Often,  individually  base-lined  programs 

reflects  both  the  hardware  and 

and  their  various  prime  or  GFE  elements  are  actually  part  of  a  SoS  approach.  The  overall  parent 

the  software  product  structure? 

program  -  the  SoS  or  joint  program,  needs  to  be  identified  with  the  various  child  programs.  Each 
child  program  would  develop  a  stand-alone  WBS  structure  [3. 3. 2. Cl] 

3.3.2.C9:  The  Contract  WBS  (CWBS)  is  the  complete  WBS  as  included  in  the  DoD-approved  PWBS 
extended  to  the  agreed-to  contract  reporting  level  and  any  discretionary  extensions  to  lower 
levels  for  reporting  or  other  purposes.  It  adequately  defines  the  lower  level  components  of  what 
is  to  be  procured  and  includes  all  the  product  elements  (hardware,  software,  data,  or  services), 
which  are  defined  by  the  contractor. 
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3.3 

Use  of  prototypes,  exercises. 

3.1.1 

Acquisition 

3.1.1.C3:  The  Acquisition  Strategy  documents  the  ground  rules  and  assumptions  under  which  the 

models,  and  simulations  to 

3.2 

Strategy/Credibility 

program  was  started  and  upon  which  future  decisions  will  be  gauged.  It  becomes  more  definitive 

determine  technological  solution 

4.2.3 

Knowledge-Based 

over  the  execution  of  the  program  in  describing  the  relationships  of  the  following  essential 

maturity 

Decisions  and 

elements: 

Milestones 

-Simulation-Based  Acquisition  (SBA)  -  Acquisition  strategy  should  address  SBA,  the  robust  and 

Technology 

interactive  use  of  modeling  and  simulation  (M&S)  throughout  the  product  life  cycle. 

Maturity  and 

3.1.1.Q33:  How  does  the  Acquisition  Strategy  describe  the  PM's  use  of  Simulation-Based 

Integration 

Acquisition  (SBA)  throughout  the  product  life  cycle?  Note:  The  PM  should  use  SBA  and  M&S 
during  system  design,  system  T&E,  and  system  modification  and  upgrade.  In  collaboration  with 
industry  and  operational  users,  PMs  should  integrate  SBA/M&S  into  program  planning  activities; 
should  plan  for  life  cycle  application,  support,  documentation,  and  reuse  of  models  and 
simulations;  and  should  integrate  SBA/M&S  across  the  functional  disciplines  [3.1.1.C3] 

•  Design  Readiness  Review.  Knowledge  should  indicate  that  the  product  can  be  built  consistent 
with  cost,  schedule,  and  performance  parameters.  This  means  design  stability 
and  the  expectation  of  developing  one  or  more  workable  prototypes  or  engineering 
development  models. 
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3.  Will  risky  new  technology  mature 
3  before  Milestone  B?  Is  there  a 
^  risk  mitigation  plan? 


3.  Have  the  key  non-technical  risk 
3  drivers  been  identified  and 
2  covered  by  risk  mitigation  plans? 
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References* 

*Not  every  reference  is  recorded  here. 

4.2. 3.  C4:  The  results  of  a  demonstration/validation  of  new  or  advanced  technologies  quantify  risk 
elements,  and  support  the  design  strategy.  A  risk  mitigation  plan  is  initially  developed  to  address 
the  attendant  risks,  including  adequate  resources  and  schedule  to  accomplish  planned  mitigation 
activities. 

4.2. 3.  Q7:  What  is  the  plan  for  the  demonstration  and  validation  of  the  proposed  technologies 
and  the  quantifiable  risks  that  remain  to  mature  the  technologies  for  system  development  and 
integration? 

•What  are  the  risk  mitigation  plan  and  the  resources  required  to  validate  (i.e.,  verification  testing, 
modeling  and  simulation,  etc)?  [4.2. 3. C4] 

Pre  milestone  B-- 

4.2. 3.  C7:  The  SE  process  manages  technology  maturation  within  the  context  of  the  documented 
Technology  Development  Strategy  (TDS),  and  manages  the  associated  risk. 

4.2. 3.  C8:  Fiscal  Year  2006,  Public  Law  109-163,  Section  801  requires  that  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology  and  Logistics  (USD(AT&L))  certify,  before  Milestone  B,  that 
"the  technology  in  the  program  has  been  demonstrated  in  a  relevant  environment."  This  wording 
immature  critical  technology,  a  more  mature  alternative  technology  has  been  identified  in 
order  to  reduce  the  program  risk  if  the  immature  technology  does  not  mature  as  planned. 

This  is  described  in  the  Critical  Technology  Element  (CTE)  maturation  plan,  which 
explains  in  detail  how  the  required  TRL  will  be  reached  prior  to  the  next  milestone  decision 
date  or  relevant  decision  point.  This  plan  includes  the  identification  of  adequate  resources 
and  schedule  to  accomplish  planned  mitigation  activities. 

The  following  knowledge  points  coincide  with  decisions  along  the  acquisition  framework: 

•  Program  Initiation.  Knowledge  should  indicate  a  match  between  the  needed  capability  and 
available  resources  before  a  program  starts.  In  this  sense,  resources  is  defined  broadly,  to  include 
technology,  time,  and  funding. 

•  Design  Readiness  Review.  Knowledge  should  indicate  that  the  product  can  be  built  consistent 
with  cost,  schedule,  and  performance  parameters. 

The  SRR  is  intended  to  confirm  that  the  user's  requirements  have  been  translated  into  system- 
specific  technological  requirements,  that  critical  technologies  are  identified,  required  technology 
demonstrations  are  planned,  risks  are  well  understood,  and  mitigation  plans  are  in  place 
[3.2.2.C7] 
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3~ 

Is  there  a  sufficient  collection  of 

1.1.1 

Mission  Description 

1.1.1.Q17:  Is  there  traceability  among  the  CONOPS,  the  capabilities/requirements  generation 

3. 

models,  and  appropriate 

1.3 

Capabilities 

process,  and  system  performance  parameters  to  validate  the  end  product  through  test  and 

3 

simulation  and  exercise 

evaluation  (T&E)?  [1.1.1.C7] 

environments,  to  validate  the 

Capabilities  Development  Document  (CDD).  The  CDD  replaced  the  Operational  Requirements 

selected  concept  and  the 

Document. 

CONOPS  against  the  KPPs? 

The  CDD  will  be  validated  and  approved  before  Milestone  B.  The  CDD  will  be  validated  and 
approved  prior  to  program  initiation  for  shipbuilding  programs.  The  CDD  provides  the  operational 
performance  attributes  necessary  for  the  acquisition  community  to  design  a  proposed  system(s) 
and  establish  a  program  baseline.  It  identifies  the  performance  attributes,  including  Key 
Performance  Parameters  (KPPs),  that  guide  the  development  and  demonstration  of  the  proposed 
system. 

Capability  Production  Document  (CPD).  The  CPD  is  the  sponsor's  primary  means  of  providing 
authoritative,  testable  capabilities  for  the  Production  and  Deployment  phase.  A  CPD  is  finalized 
after  the  design  readiness  review  and  must  be  validated  and  approved  before  the  Milestone  C 
acquisition  decision.The  CPD  refines  the  threshold  and  objective 
values  for  the  performance  attributes  and  KPPs  that  are  validated  in  the  CDD  for  the 
production  phase.  The  refinement  of  performance  attributes  and  KPPs  is  the  most 
significant  difference  between  the  CDD  and  CPD. 
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3. 

Has  the  claimed  degree  of  reuse 

3.3.4 

Management 

3.3.4.3.C8:  The  Government  Program  Office  should  initially  approve  the  program  metrics  and 

3. 

been  validated? 

4.5.1 

Methods,  Metrics, 

then  periodically,  e.g.,  monthly,  the  metrics  should  be  reported  and  reviewed.  These  metrics 

4 

and  Techniques 

should  include  many,  if  not  all  of  the  following:  Development  status  S  curves;  Processor 

Software 

throughput  utilization;  Processor  memory  utilization;  Input/output  utilization;  Software 

Development  Plan 

Engineering  Staffing;  Software  Work  Packages  Summary;  Schedule  Performance  Index;  Cost 
performance  Index;  Problem/Deficiencies /Discrepancies  Status;  Requirements  Stability;  Software 
Size;  Software  Reuse  Status  (planned  versus  'actuals');  Reliability  Growth  Curve;  Logistics 
Footprint  Reduction;  Planned  Operational  Effectiveness;  Product  Availability  Predictions;  O&S 

Cost  Projections;  Development  Test  entrance  criteria  and  status;  DAES  Reporting  (For  MDAPS); 
Milestone  B  and  C  entrance  criteria. 

4.5.1.C17:  Reuse  of  software,  from  existing  systems  or  prior  development  efforts,  has  been 
analyzed  for  complexity  and  suitability  to  meet  required  functionality,  in  accordance  with 
accepted  software  engineering  standards.  Pre-Milestone  C,  this  analysis 
has  resulted  in  documented  re-use  in  line  with  plans. 
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3.4 

Validated  system  engineering. 

3.3.1 

Program 

•  Integrated  Baseline  Review  (IBR)  -  The  IMS  facilitates  the  conduct  of  a  successful  IBR,  in  which  it 

development,  manufacturing. 

Plan/Schedule 

is  verified  there  is  sound  basis  for  cost  and  schedule  execution  of  program  objectives,  program 

operations  &  maintenance 

risks  are  addressed  and  the  contractor  has  performance  plans  and  underlying  management 

budgets  and  schedules 

control  systems  to  assess  the  realism  of  the  performance  measurement  baseline  providing  the 
required  baselines. 

105 
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3~ 

4. 

1 

Are  the  major  known  cost  and 
schedule  drivers  and  risks 
explicitly  identified,  and  is  there 
a  plan  to  track  and  reduce 
uncertainty? 

2.1 

2.2.1 

3.1.2 

3.2.2 

Resources/Program 
Schedule  Overview 
(TIER  1) 

Program  Funding 
and  Allocation 
Acquisition 

Strategy 

Entrance  and  Exit 

Criteria 

Perspective:  Experienced  program  personnel  provide  data  regarding  critical  and  high-risk  efforts 
and  identify  as  realistically  as  possible  the  expected  schedule,  which  the  program  management 
office  then  compares  with  the  top-level  defense  program  schedule  template  to  determine  the 
actual  schedule  risk  and  to  identify  all  schedule  drivers.  With  this  approach,  the  probability  of 
overrunning  a  program  schedule  can  be  estimated  by  determining  how  much  risk  exists  and 
where  it  is  greatest.  This  approach  enables  program  managers  (PMs)  to  estimate  early  and 
continuously  in  the  program  the  possibility  of  a  significant  likelihood  of  overrunning  the  program 
schedule  by  determining  how  much  and  where  the  risk  to  successful  schedule  completion  is 
greatest. 

2.2.1. Q13:  Is  the  program,  as  captured  in  the  CARD-like  document,  executable? 

•  Does  the  CARD-like  document  capture  the  key  program  cost  drivers,  development  costs  (all 
aspects  of  hardware,  human  integration,  and  software),  production  costs,  and  operation  and 
support  costs? 

3.1.2. C5:  The  Acquisition  Strategy  and  specific  acquisition  approaches  are  consistent  with 
operational  capabilities/requirements  and  available  resources,  and  appropriate  to  fully 
develop  a  system  that  meets  the  program  objectives. 

•  Program  risks  are  identified  and  documented,  and  progress  is  tracked  via  established 
metrics  that  should  be  invariant  with  time.  The  end  result  is  the  overall  risk  of  implementing 
the  Acquisition  Strategy  is  considered  to  be  manageable  within  available  time  and 

resources. 

3.2.2. Q48:  Did  the  following  result  from  the  SRR? 

•  A  comprehensive  risk  assessment  for  the  SDD  phase 

•  An  approved  SDD  SEP  that  addresses  cost  and  critical  path  drivers 
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3. 

4. 

2 

Have  the  cost  confidence  levels 
been  developed  and  accepted  by 
the  key  system  stakeholders? 

2.2.2 

Continuity  and 
Stability 

Assuming  acceptance  is 
implied  because  it  was 
created  with  the 
contractors/stakeholde 
rs  involved 

pre  all  milestones: 

2. 2. 2. Cl:  Flow  of  funding  is  stable  and  steady  throughout  the  phases  of  the  system's  acquisition 
life  cycle.  The  program  manager  (PM)  and  contractor  plan  for  perturbations  in  the  budget,  both 
from  within  and  outside  their  spectrum  of  control.  Accordingly,  the  PM  has  taken  the  following 
minimal  steps  to  achieve  greater  control  over  maintaining  a  stable  budget:  obtaining  a  high- 
confidence  cost  estimate  that  is  well  documented  to  firmly  support  budget  requests;  ensuring 
user  advocacy  for  the  program;  ensuring  that  funding  for  the  execution  year(s)  is  consistent  with 
the  contractor's  ability  to  expend  the  funding  according  to  the  current  program  schedule;  and 
developing  a  range  of  independent  estimates  at  completion  from  earned  value  data  and  analysis 
of  the  integrated  master  schedule.  Compare  the  results  with  the  contractor's  projected  final  costs 
to  assess  realism  and  to  form  the  basis  for  adjusting  the  program  budget. 
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3. 

4. 

3 

Is  there  a  top-to-bottom  plan  for 
how  the  total  system  will  be 
integrated  and  tested?  Does  it 
adequately  consider  integration 
facilities  development  and  earlier 
integration  testing? 

1.3.1 

4.6.1 

Reasonableness, 
Stability,  and 
Testability 

Test  and  Evaluation 

Plan 

The  CPD  captures  the  information  necessary  to  support  production,  testing,  and  deployment  of 
an  affordable  and  supportable  system  within  an  Acquisition  Strategy.  The  CPD  refines  the 
threshold  and  objective  values  for  the  performance  attributes  and  KPPs  that  are  validated  in  the 
CDD  for  the  production  phase. 

4.6.1.  Cl:  The  program  manager  (PM)  shall  develop  a  robust  integrated  Test  and  Evaluation 
Strategy  (TES)  for  all  phases  of  the  program,  describing  developmental  test  and  evaluation 
(DT&E),  operational  test  and  evaluation  (OT&E),  and  live-fire  test  and  evaluation  (LFT&E). 

Without  compromising  rigor,  the  program  is  required  to  integrate  modeling  and  simulation 
(M&S)  activities  into  the  strategy.  The  TES  should  be  consistent  with  and  complementary  to  the 
Systems  Engineering  Plan  (SEP). 

4.6.1. C3:  The  system  integration,  test  and  evaluation  process  is  defined  in  the  Technology 
Development  Strategy  (TDS)  and  includes  analysis,  reviews,  inspections,  demonstrations,  testing, 
and  M&S  to  evaluate  the  requirements  baseline  and  the  system's  progress  during  development 
to  meet  the  critical  technical  parameters  (CTPs).  The  TES  describes  an 

iterative  process  by  which  allocated  specifications  and  CTPs  are  met  by  lower-level 
components,  assemblies,  subsystems  and  then  at  the  system  level.  Requirements 
are  traceable  to  specific  test  and  evaluation  events. 

4.6.1. C7:  Integration  test  facilities  that  allow  demonstration  of  hardware  and  software 
operation  at  progressively  higher  levels  of  integration  are  used/planned  during  TD. 
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3. 

4. 

4 

If  time-boxing  or  time- 
determined  development  is  used 
to  stabilize  schedules,  have 
features  been  prioritized  and  the 
system  architected  for  ease  of 
adding  or  dropping  borderline 
features? 

4.2.2 

1.3 

2.2 

Management  of 
Requirements 
Capabilities 
Constraints  and 
Dependencies 

DAPS  does  not  address 
"prioritized" 
requirements  but 
requires  a  fexible 
process  that  supports 
change  when  needed. 

It  also  recommends  a 
modular  and  open 
architecture  to 
facilitate  change. 

Time-boxing  or  time- 
determined 
development  is  not 
mentioned,  but  it 

mentions 
accomodating 
constraints  of  time.  It 

also  recommends  a 

time  reserve. 

4.2. 2. CIO:  The  evolutionary  Acquisition  Strategy  (AS)  utilizes  a  management  system  that 
continually  defines  the  requirements  and  development  activities  to  support  the  evolving  needs; 
adequately  addresses  the  various  concerns  of  users,  developers,  and  managers;  and  mitigates  the 
risks  associated  with  these  issues.  The  basic  system  architecture  is  designed  to  accommodate 
change.  Techniques  such  as  open  systems  design,  functional  partitioning  and  modular  design 
have  been  addressed  by  the  PM  to  achieve  a  flexible  system  that  can  be  easily  and  affordably 
modified. 

1.3:  New  capabilities  are  defined  within  the  "art  of  the  possible"  and  grounded  within  real-world 
constraints  of  time,  technology,  and  affordability. 

1.3.1.  Cl:  Milestone  A  review  . . .  The  ICD  clearly  states  required  capabilities  in  broad  and  time- 
phased  operational  goals. 

2. 1.2.  Cl.  .  .The  end  result  is  a  program  schedule  that  has  inherent  flexibility  to  accommodate  the 
competing  demands  of  time  and  resources  while  ensuring  the  best  capability  to  the  warfighter. 
Note:  Constraints  are  effectively  global  requirements,  such  as  limited  development  resources  or  a 
way  a  system  is  developed.  Constraints  can  be  economic,  political,  technical,  or 
environmental  and  pertain  to  program  resources,  schedule,  environment,  or  to  the 

system  itself. 
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3. 

4. 

5 

Are  there  strategies  and  plans  for 
evolving  the  architecture  while 
stabilizing  development  and 
providing  continuity  of  services? 

4.1.2 

4.2.2 

Modular  Open 
Systems  Approach 
Management  of 
Requirements 

4.1. 2.  Cl:  Certain  capabilities,  requirements,  and  program  strategies/objectives  necessitate 
implementing  open  systems  and  developing  open  architectures.  DoDD  5000.1  requires  that  a 
modular  open  systems  approach  (MOSA)  be  employed  where  feasible.  The  program  should 
identify  open  architecture  enabled  capabilities/objectives  that  reflect  the  following  MOSA 
objectives  (see  the  MOSA  PM  Guide  at  (http://www.acq.osd.mil/osjtf/pmguide.html): 

1.  Facilitate  a  modular  architecture  to  allow  for  affordable  interoperability 

2.  Ensure  a  flexible  and  robust  system  design  to  accommodate  changing  technology  and 
requirements 

3.  Facilitate  integration  with  other  systems  and  use  of  commercial  products  from  multiple 
sources  both  in  the  initial  design  and  in  future  enhancements 

4.  Enable  technology  insertion  as  currently  available  commercial  products  mature  and  new 
commercial  products  become  available  in  the  future 

4.2. 2.  CIO:  The  evolutionary  Acquisition  Strategy  (AS)  utilizes  a  management  system  that 
continually  defines  the  requirements  and  development  activities  to  support  the  evolving  needs; 
adequately  addresses  the  various  concerns  of  users,  developers,  and  managers; 

and  mitigates  the  risks  associated  with  these  issues.  The  basic  system  architecture 
is  designed  to  accommodate  change.  Techniques  such  as  open  systems  design,  functional 
partitioning  and  modular  design  have  been  addressed  by  the  PM  to  achieve  a  flexible 
system  that  can  be  easily  and  affordably  modified. 
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4  Evidence-based  progress  monitoring 
and  commitment  reviews 

3.3.4  Management  Sub-Factor  3. 3. 4. 3 -Technical  Performance  Measures 

Methods,  Metrics,  Pre-Milestone  B  &  C 

4.3.1  and  Techniques  Criteria 

Technical  Review  3. 3. 4.3. Cl:  Systems  engineering  uses  technical  performance  measurements  to  balance  cost. 

Planning  schedule,  and  performance  throughout  the  life  cycle.  Technical  performance  measurements 

compare  actual  versus  planned  technical  development  and  design.  They  also  report  the  degree  to 
which  system  requirements  are  met  in  terms  of  performance,  cost,  schedule,  and  progress  in 
implementing  risk  handling.  Performance  metrics  are  traceable  to  user-defined  capabilities. 

Factor  4.3.1  -  Technical  Review  Planning 

All  Acquisition  Category  (ACAT)  programs  should  include  the  essential  technical  reviews  shown  on 
the  timeline,  as  applicable.  Technical  reviews  provide  a  systematic  process  for  continuously 
assessing  the  technical  baseline,  design  maturity,  technical  risk,  and  programmatic  risk  of 
acquisition  programs.  Technical  reviews  are  consistent  with  existing  and  emerging  commercial 
and  industrial  standards  and  form  the  backbone  of  an  effective  Systems  Engineering  Plan  (SEP). 
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4.1 

Monitoring  of  system  definition 
&  development  progress  vs. 
plans 

2.1 

Program  Schedule 
Overview 

"As  the  program  progresses,  the  PM  monitors  the  effectiveness  of  handling  activities  included  in 
the  integrated  planning  events  and  schedule  by  comparing  observed  activity  results  with  their 
criteria  and  determining  any  deviations  from  the  planned  schedule." 
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4. 

Are  the  levels  and  formality  of 

3.3.4 

Management 

Not  clear  what  "level  of 

3. 3. 4.2. Cl:  EVM  is  required  on  all  cost  or  incentive  type  acquisition  contracts,  subcontracts,  intra- 

1. 

plans,  metrics,  evaluation  criteria. 

Methods,  metrics. 

project  requirements 

government  work  agreements,  and  other  agreements  according  to  dollar  thresholds  prescribed  in 

1 

and  associated  mechanisms  (IMP, 

and  Techniques 

emergence"  means. 

USD(AT&L)  Policy  Memorandum  dated  March  7,  2005.  The  thresholds  are  as  follows: 

IMS,  WBS,  EVMS)  commensurate 

Needs  clarification. 

•$20  million  or  greater  -  EVM  implementation  compliant  with  ANSI/EIA  -  748  -  A  is  required.  No 

with  the  level  of  project 

formal  EVM  System  (EVMS)  validation  is  required 

requirements  emergence  and 

In  DAPS,  levels  of 

•$50  million  or  greater  -  EVM  implementation  compliant  with  ANSI/EIA  -  748  -  A  is  required.  An 

stability?  (Too  little  is  risky  for  pre 

formality  specified  by 

EVM  System  must  be  formally  validated  and  accepted  by  the  cognizant  contracting  officer 

specifiable  and  stable 

size  of  project,  not 

•  A  Contract  Performance  Report  (CPR)  and  Integrated  Master  Schedule  (IMS)  are  required 

requirements;  too  much  is  risky 

necessarily  stability  of 

deliverables  for  all  contracts  that  are  $20  million  or  greater  that  require  EVM 

for  emergent  and  unstable 
requirements.) 

requirements. 

•  Less  than  $20  million  -  EVM  is  not  required,  except  at  the  discretion  of  the  PM 
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4. 

Are  the  project's  staffing  plans 

2.1.1 

Program 

Not  sure  what  "buildup 

2. 1.1. Cl: . .  .The  program  manager  (PM)  has  utilized  subject  matter  expertise  of  the  stakeholders 

1. 

and  buildup  for  progress 

2.3.1 

Schedule/Viability 

for  progress 

and  the  following  processes  to  develop  the  program  schedule: 

2 

monitoring  adequate  with 

Sufficiency  of 

monitoring"  means. 

2.1.1.Q26:  What  are  the  changes  in  the  personal  experience  and  subject  matter  expertise  of  the 

respect  to  required  levels  of 

Numbers  and 

IPT  members  involved  in  the  development  of  the  program  schedule? 

expertise? 

Qualifications 

2.3.1. Q1:  Is  there  a  staffing  plan  established? 

•  What  is  the  process  to  determine  personnel  resources  and  phasing  required  for  the 
development  of  the  staff,  including  skills,  experience,  and  education  level? 

•  What  are  the  metrics  and  standards  used  to  measure  the  quality  of  the  workforce?  [2.3.1. Cl] 

2.3.1. Q18:  What  is  the  experience  level  of  each  of  the  existing  or  planned  key  technical 
personnel? 

•  What  engineering  expertise  is  required  for  the  program? 

•  How  is  the  experience  of  technical  personnel  relevant  to  the  current  activity?  [2.3.1.C3] 
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4. 

Have  most  of  the  planned  project 

2.3.1 

Sufficiency  of 

2.3.1.Q3:  How  does  the  PMO  describe  the  personnel  issues  affecting  the  program's  ability  to 

1. 

personnel  billets  been  filled  with 

Numbers  and 

successfully  execute  the  program? 

3 

staff  possessing  at  least  the 

Qualifications 

•  What  key  specialties  are  missing? 

required  qualification  level? 

•  What  key  billets  are  unfilled/about  to  be  vacated?  [2. 3.1. Cl] 

2.3.1.Q4:  What  is  the  experience  level  of  each  of  the  existing  or  planned  key  technical  personnel? 
How  is  the  experience  of  technical  personnel  relevant  to  the  current  activity?  [2.3.1. Cl] 

123 

4. 

Is  the  project  adequately 

3.3.4 

Management 

Sub-Factor  3.3.4. 1  -  Risk  Management 

1. 

identifying  and  managing  its 

Methods,  metrics. 

3. 3. 4.1. Cl:  The  Department  of  Defense  (DoD)  recognizes  that  risk  management  is  critical  to 

4 

risks? 

and  Techniques 

acquisition  program  success  (see  the  Defense  Acquisition  Guidebook  (DAG). 

3.3.4.1.C2:  There  are  several  notable  changes  of  emphasis  in  the  above  guide  from  previous  RM 
versions.  These  changes  reflect  lessons  learned  from  application  of  risk  management  in  DoD 
programs.  Emphasis  has  been  placed  on: 

•  The  role  and  management  of  future  root  causes, 

•  Distinguishing  between  risk  management  and  issue  management, 

•Tying  risk  likelihood  to  the  root  cause  rather  than  the  consequence, 

•Tracking  the  status  of  risk  mitigation  implementation  versus  risk  tracking,  and 

•  Focusing  on  event-driven  technical  reviews  to  help  identify  risk  areas  and  the  effectiveness  of 
ongoing  risk  mitigation  efforts. 
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SE  EM  Framework  Area 


125  4.  Have  the  processes  for 

1  conducting  reviews  been 
5  evaluated  for  feasibility, 

reasonableness,  completeness, 
and  assurance  of  independence? 


126  4.  Has  compliance  with  legal,  policy, 

1  regulatory,  standards,  and 
g  security  requirements  been 
clearly  demonstrated? 


127  _ 

128  4.2  Monitoring  of  feasibility 

evidence  development  progress 
vs.  plans 


Interpretation  DAPS  DAPS  Topic 
_ Section _ Covered 

4.3.1  Technical  Review 

1.3.1  Planning 
Reasonableness, 

4.1.1  Stability,  and 
Testability 
System  Assurance 


3.2.1  Statutory  and 
Regulatory 

3.2.3  Compliance  and 
3.4  Guidance 

Certifications 

Contract 

Management 

2.1.1  Viability 

3.2.1  Statutory  and 
Regulatory 
Compliance  and 
Guidance 


Comments/ 

Observations 


References* 

*Not  every  reference  is  recorded  here. 


States  that  the  reviews 
are  held,  "as 
applicable" 

Independent 
assessments  are 
performed  throughout, 
but  the  contractor's 
processes  for 
conducting  reviews  is 
not  really  addressed 
except  by  the  SEP. 


This  question  seems 
too  vague.  It  assumes 
a  lot  of  knowledge  and 
can  pertain  to 
reporting  mechanisms, 
processes,  government 
and  contract  conduct. 


Factor  4.3.1 -Technical  Review  Planning 

All  Acquisition  Category  (ACAT)  programs  should  include  the  essential  technical  reviews  shown  on  the 
timeline,  as  applicable.  Technical  reviews  provide  a  systematic  process  for  continuously  assessing  the 
technical  baseline,  design  maturity,  technical  risk,  and  programmatic  risk  of  acquisition  programs. 

1.3.1. C14:  Computer/software  configuration  items  have  completed  test  verification,  and  the  system 
software  capability  is  determined  to  be  mature.  All  known  deficiencies  have  been  documented  and 
evaluated,  and  fixes  have  been  identified  and  rescheduled  for  verification.  An  Independent  Verification  and 
Validation  (IV&V)  assessment  of  the  contractor/materiel  developer  has  been  performed. 

4.3.1. C3:  The  ITR  ensures  that  a  program's  technical  baseline  is  sufficiently  rigorous  to  support  a  valid  cost 
estimate  (with  acceptable  cost  risk),  and  enable  an  independent  assessment  of  that  estimate  by  cost, 
technical,  and  program  management  subject  matter  experts. 

4.3.1. C15:  The  SRR  is  typically  conducted  by  a  technical  review  board  consisting  of  a 
government  chairperson  selected  outside  (independent  of)  the  government  program 
office. 

4.3.1. Q32:  For  ACAT  ID  or  1AM  programs,  the  service  acquisition  official  provides  a 
recommendation  to  the  Director,  Defense  Research  and  Engineering  (DDR&E)  of  the 
Office  of  the  Secretary  of  Defense  for  Deputy  Under  Secretary  of  Defense  for  Science 
and  Technology  (DUSD(S&T))  final  approval.  If  deemed  necessary,  the  DDR&E  can 
conduct  an  Independent  Technical  Assessment  (ITA)  in  addition  to,  and  totally  separate 
from,  the  TRA: 

4.3.1. C56:  Prior  to  the  OTRR  the  OUSD(AT&L)  Systems  and  Software  Engineering/ 

Assessments  and  Support  (SSE/AS)  staff  will  conduct  an  assessment  of  operational 

test  readiness  (AOTR)  to  independently  assess  the  successful  completion  of 

developmental  test  and  evaluation  (DT&E)  and  report  the  AOTR  findings  to  the  PM  and  Deputy, 

OUSD(A&T). 

4.4.2.  C7:  Government  and  contractor  use  common  M&S  tools  to  support  both  development  and  test 
and  evaluation.  Simulations  used  to  evaluate  program  performance  as  part  of  the  test  and  evaluation 
process  are  verified  independently  from  contractor  simulations  and  undergo  the  same  level  of 
verification,  validation,  and  accreditation  (VV&A). 

4.5. 2.  Q3:  How  will  the  various  program  estimates  be  vetted?  How  similar/different  are  the  program's 
cost  and  schedule  estimates  and  associated  assumptions  to  other  estimates  (e.g.  the  Cost  Analysis 
and  Information  Group's  (CAIG)  independent  cost  estimate  (ICE))? 

3.2.1. Q10:  Does  the  PMO  have  a  clear  and  concise  understanding  of  all  DoD  and  Service-level 
policies  and  statutes  that  the  program  must  comply  with?  [3.2.1.C4] 

3.2.1. Q11:  Have  the  following  statutory  information  requirements  been  met?  Who  is  the 
approval  authority  and  what  is  the  approval  date?  Note:  See  DAG  for  applicable  statutes  for  each 
information  requirement. 

•  Consideration  of  technology  issues 


2.1.1.Q9:  What  is  the  process  established  to  monitor  program  performance  through  the 
schedule? 

•Are  the  following  identified? 

-Key  events 
-  Milestones 
-Reviews 

-All  integrated  technical  tasks 

-Accomplishment  criteria  and  schedule  metrics  [2.1.1.C2] 
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Comments/ 

Observations 

2.1.1. Q17:  What  is  the  highest  risk  path,  both  for  the  overall  program  schedule  and  for  the  SDD 
schedule? 

•  How  has  the  PM  applied  resources  against  the  activities  on  this  risk  path?  [2.1.1.C3  and  2.1.1.C4] 

3.2.2. Q32:  In  preparation  for  the  SRR,  were  the  following  actions  completed? 

•  Successful  completion  of  all  post-award  activities 

•  Published  agenda  (several  weeks  prior  to  the  conference  -  to  permit  sufficient  time  for 
government  preparation 

•  Draft  system  specification  and  any  initial  draft  performance  item  specifications 

•  Functional  analysis  (top  level  block  diagrams) 

•  Feasibility  analysis  (results  of  technology  assessments  and  trade  studies  to  justify  system  design 
approach) 

3. 3. 4.1. Cl:  The  Department  of  Defense  (DoD)  recognizes  that  risk  management  is  critical  to 
acquisition  program  success  (see  the  Defense  Acquisition  Guidebook  (DAG).  The  purpose  of 
addressing  risk  on  programs  is  to  help  ensure  program  cost,  schedule,  and  performance 
objectives  are  achieved  at  every  stage  in  the  life  cycle  and  to  communicate  to  all  stakeholders  the 
process  for  uncovering,  determining  the  scope  of,  and  managing 
program  uncertainties.  Since  risk  can  be  associated  with  all  aspects  of  a  program,  it  is 
important  to  recognize  that  risk  identification  is  part  of  the  job  of  everyone  and  not  just 
the  program  manager  or  systems  engineer.  That  includes  the  test  manager,  financial 
manager,  contracting  officer,  logistician,  and  every  other  team  member. 


References* 

*Not  every  reference  is  recorded  here. 
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J7 

Has  the  project  analyzed 

1.2.1 

Validity  and 

1.2.1.Q1:  How  were  mission  tasks  (MTs),  measures  of  effectiveness  (MOEs),  and  measures  of 

2. 

alternative  methods  of  evaluating 

1.2.2 

Currency 

performance  (MOPs)  derived  from  relevant  guidance  on  requirements  or  capabilities  (e.g.. 

2 

feasibility  (models,  simulations. 

1.3 

Linkage  and 

Mission  Needs  Statement  (MNS),  Operational  Requirements  Document  (ORD)  (if  pertinent),  or 

benchmarks,  prototypes. 

2.1.1 

Traceability 

the  problem  statement  found  in  the  ICD?  [1.2.1. Cl] 

reference  checking,  past 

3.4.3 

Capabilities 

•Are  they  quantifiable?  [1.2.1. Cl] 

performance,  etc.)  and  prepared 

4.2.4 

Program 

1.2.1.Q7:  What  are  the  models  and  simulations  used  in  the  study? 

the  infrastructure  for  using  the 

Schedule/Viability 

•Are  they  acceptable  and  accredited?  [1.2.1. Cl] 

most  cost-effective  choices? 

Value  Engineering 

1.2. 2. Cl:  The  Analysis  of  Alternatives  (AoA)  study  plan  describes  a  clear  link  between  the  AoA, 

Trade  Studies  and 

capability  needs,  system  requirements,  and  the  measures  of  effectiveness  (MOEs)  used  to 

Approches 

evaluate  the  system(s). 

1.2.2.Q13:  How  does  the  program  use  realistic  and  current  architectures  and  the  CONOPS  to 
derive  alternative  solutions  and  to  ensure  a  clear  understanding  of  potential  C4I  interfaces  and 
interoperability  needed  during  military  operations? 

•  Initial  Capabilities  Document  (ICD) . . .  supports  the  Analysis  of  Alternatives  (AoA),  Technology 
Development  Strategy  (TDS),  Milestone  A  decisions,  and  subsequent  Technical  Development  (TD) 
phase  activities. 

2.2.1. Q14:  What  were  the  results  of  the  Alternative  System  Review  (ASR)? 

•  Did  the  IPT  determine  that  the  operational  capabilities,  preferred  solution(s),  available 
technologies,  and  program  resources  (funding,  schedule,  staffing,  and  processes)  form 

a  satisfactory  basis  for  proceeding  into  the  TD  phase? 

•Is  the  program  schedule  executable  (technical  and/or  cost  risks)?  [2.2.1.C2] 

3.1.1. Q23:  How  did  the  Acquisition  Strategy  address  risk  management? 

•What  statistical  or  other  qualitative  procedures  were  followed  to  "measure"  program 
risk? 

•What  is  the  risk  management  structure  for  selecting  acquisition  alternatives?  [3.1.1.C3] 

3.4.3. Q23:  How  did  the  government  and  contractor,  through  the  VE  process,  analyze  the 
essential  requirements,  military  and  technical  characteristics,  and  the  design  tasks  to 
develop  possible  alternatives  offering  improved  value? 

4.2.4. Q8:  What  is  the  planned  role  of  modeling  and  simulation  (M&S)  to  support  trade  studies? 
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1.2.1  Validity  and 
4.1.8  Currency 

4.4.2  Human  Systems 
Integration 
Modeling  and 
Simulatioin  Tools 


Program 

Plan/Schedule 


References* 

*Not  every  reference  is  recorded  here. 

1.2.1. Q1:  How  were  mission  tasks  (MTs),  measures  of  effectiveness  (MOEs),  and  measures  of 
performance  (MOPs)  derived  from  relevant  guidance  on  requirements  or  capabilities  (e.g.. 

Mission  Needs  Statement  (MNS),  Operational  Requirements  Document  (ORD)  (if  pertinent),  or 
the  problem  statement  found  in  the  ICD?  [1.2.1. Cl] 

•Are  they  quantifiable?  [1.2.1. Cl] 

1.2.1. Q2:  Are  the  MOEs  stated  in  terms  of  military  utility  and  based  on  value  provided  to  the 
warfighter? 

•Are  these  MOEs  used  to  identify  models,  simulations,  and  other  analysis  tools  required  to 
execute  the  study?  [1.2.1. Cl] 

1.2.1. Q3:  What  are  the  relevant  issues  and  constraints  as  addressed  in  the  study  plan?  [1.2.1. Cl] 

1.2.1. Q4:  Is  the  range  of  alternatives  comprehensive?  [1.2.1. Cl] 

1.2.1. Q5:  Are  the  threats  and  scenarios  realistic  and  current?  [1.2.1. Cl] 

1.2.1. Q15:  Were  the  threats  and  scenarios  used  in  the  study  appropriate  and  approved  by 
Defense  Intelligence  Agency  (DIA)? 

4.1.8:  •  Are  scenario-based  factors  such  as  environmental  conditions,  conflict  durations,  etc. 
included?  [4.1.8.C1,  C2] 

4.4.2.  C5:  The  program  has  a  plan  to  acquire  domain  knowledge  for  each  M&S  objective- 
scenario  set.  This  domain  knowledge  includes  the  entities,  attributes  and  interactions 
that  have  significant  bearing  on  the  objective  at  the  level  of  resolution  and  fidelity 
required  for  the  effort. 

Unclear  if  this  means  3.3.1.C7:  The  program  has  appropriate  development  activities  planned  and  scheduled,  e.g. 
establishing  thresholds  Integrated  Master  Plan/Integrated  Master  Schedule  (IMP/IMS),  and  implements  these  activities 
for  the  schedule  and  to  execute  the  program.  These  planned  and  scheduled  activities  include  completion  criteria, 
cost  metrics.  Target  Program  funding  and  schedules  are  sufficient  to  accommodate  technical  complexity  and 

values  are  not  identified  program  risks.  Sufficient  resources  are  allocated  and  available  to  the  program  to 

discussed  in  DAPS  successfully  develop  the  system  within  the  program  baseline. 

3.3.1.C8:  The  program  is  following  the  program  management  plans  in  executing  the  program.  The 
program  has  accomplished/is  accomplishing  the  planned  activities  with  minimal  schedule  impact 
and  is  proceeding  to  execute  within  the  program  baselines.  Schedule  performance  is  reported 
through  an  Earned  Value  Management  System  (EVMS). 


Comments/ 

Observations 
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J7 

Is  the  project  successfully 

3.4.1 

Contracting/Prime 

3.4.1.C5b:  Design  and  Production  Assurance  -  monitor  the  performance  of  the  contractor  against 

2. 

monitoring  progress  and  applying 

Contractor 

contract  requirements  to  enable  timely  corrective  action. 

5 

corrective  action  where 

4.6.1 

Management 

3.4.1.Q74:  In  terms  of  Integrated  Baseline  Reviews  (IBRs): 

necessary? 

Test  and  Evaluation 

•  How  did  the  contractor  address  the  government's  intent  to  conduct  IBRs  after  contract  award? 

Plan 

•  Who  developed  the  guidelines,  criteria,  and  processes  for  the  IBR? 

•  Who  lead  the  technical  assessments  during  IBRs? 

•  Upon  completion,  how  are  the  results  of  the  IBR  documented  and  provided  to  appropriate 
team  members? 

•  What  action  plan  is  prepared  to  correct  any  problem  areas  discovered  during  the  review? 

•What  is  the  process  to  track  corrective  actions  and  interfaces  with  the  contractor  during 
program  reviews  until  the  corrective  actions  are  completed?  [3.4.1.C5b] 

4.6.1.C11:  A  Failure  Reporting,  Analysis  and  Corrective  Action  System  (FRACAS)  has  been  initiated. 
The  systems  engineering  (SE)  process  provides  tracking  between  test  activities  and  technical 
requirements. 

•  Discuss  the  planned  FRACAS  program.  What  is  the  planned  time  for  root  cause  analysis  and 
corrective  action  for  major  and  minor  hardware/software  deficiencies? 

[4.6.1.C19,  4.6.1.C20,  and  4.6.1.C22] 
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4.3 

Monitoring,  assessment,  and 
replanning  for  changes  in  needs, 
opportunities,  and  resources 
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4~ 

3. 

1 

Does  the  project  have  an 
effective  strategy  for  performing 
triage  (accept,  defer,  reject)  on 
proposed  changes,  which  does 
not  destabilize  ongoing 
development? 

1.3.1 

2.1.1 

3.1.1 

3.2.2 

3.3.1 

3.3.6 

3.4.1 

Reasonableness, 
Stability,  and 
Testability 

Program 

Schedule/Viability 
Acquistion 
Strategy/Credibility 
Entrance  and 
Exit/Success 

Criteria 

Program 
Plan/Schedule 
management  of 
Dependencies  and 
External  Interfaces 

Prime  contractor 

Management 

DAPS  discusses 
allowing  for  and 
managing  change  but 
does  not  present 
methods  for  doing  so 
like  triage. 

1.3.1. Q8:  Were  any  changes  made  to  the  ICD  between  JROC  approval  and  the  Milestone  A  decision  review? 

•  How  were  these  changes  vetted  through  the  requirements  generation  and  acquisition  management 
processes? 

1.3.1. Q17:  What  controls  are  in  place  to  prevent  "requirements  creep"  and  to  force  new  requirements  to  be 
defined  through  an  engineering  change  proposal  process?  [1.3.1.C5] 

2. 1.1.  Cl:  The  program's  overall  schedule  is  viable  (i.e.,  workable  and  has  real  meaning  and  pertinence).. . . 

•  Implementation  of  procedure(s)  to  control  changes  to  the  program  schedule.. . . 

•  Revision  of  the  procedure(s)  to  control  changes  to  the  program  schedule,  if  required. 

2.2.1. Q4:  What  is  the  PM's  process  to  prevent  unexpected  or  unplanned  cost  growth  by  adequately 
identifying  and  managing  risks  in  the  program? 

•  What  is  the  process  to  allocate  funding  (level  and  timeliness)  to  cover: 

-  Systems  Engineering  (SE)  technical  reviews 

-  Risk  mitigation 

-  Engineering  changes 

3.1.1. 0.4:  Are  any  of  the  potential  causes  of  instability  to  the  Acquisition  Strategy  present?  If  so,  how  is  the 

PM  working  to  mitigate  the  impact  to  the  program's  Acquisition  Strategy? 

3.1.1. Q5:  How  is  the  PM  emphasizing  the  following  "aids"  to  a  stable  Acquisition  Strategy? 

•  Direction. 

•  Advocacy. 

•  Commitment. 

•  Use  of  Integrated  Product  Teams. 

3.3.1. Q9  What  are  the  in  place  processes  to  manage  the  Technology  Development  (TD)  phase  effort  and  cont 
3.3.6.Q15:  How  are  changes  in  SoS  constituent  systems  negotiated  with  their  PMs?  [3.3.6.C14] 

3.4.1. C5:  The  program's  third  phase  -  execution  and  sustainment  -  is  being  successfully  completed  through  in 

3.4.1. Q71:  How  are  Engineering  Change  Proposals  (ECPs)  and  alterations  affecting  cost  and  schedule  address^ 

•  Are  the  changes  within  the  scope  of  the  contract? 

•Was  pricing  information  to  support  the  ECP  requested  from  the  contractor? 

•After  Change  Control  Board  approval,  were  the  following  issued? 

-The  change  request  for  implementing  the  change 
-Contract  deliverable  data  requirements 
-Sole  source  authorization  if  required 
-Funding  documents  to  be  used  [3.4.1.C5b] 
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4~ 

Does  the  project  have  an 

1.3.1 

Reasonableness, 

3.2.2.Q61:  In  preparation  for  the  PRR,  were  the  following  activities/actions  completed  or 

3. 

adequate  capability  for 

Stability,  and 

documents/information  available? 

2 

performing  change  impact 

2.1.1 

Testability 

•  Provisions  have  been  made  for  determining  producibility  and  cost  impacts  of  engineering 

analysis  and  involving 

3.1.1 

Program 

changes  introduced  during  production 

appropriate  stakeholders  in 

3.2.2 

Schedule/Viability 

3.4.1.C5:  The  program's  third  phase  -  execution  and  sustainment  -  is  being  successfully 

addressing  and  prioritizing 

3.3.1 

Acquistion 

completed  through  insight  into  program  progress,  and  the  effective  management  of  the  impact 

changes? 

3.3.6 

Strategy/Credibility 

of  changes,  whether  these  changes  are  due  to  contract  execution  or  to  external  influences.  As  the 

3.4.1 

Entrance  and 

program  progresses,  the  PM  makes  viable  and  timely  decisions  and  provides  direction  to 

Exit/Success 

accommodate  changing  circumstances.  Focus  is  maintained  on  the  risk  areas  most  likely  to 

Criteria 

impact  the  program.  The  PM  uses  those  indicators  developed  in  the  previous  stages,  i.e.,  EVMS, 

Program 

IMS  and  appropriate  metrics,  for  primary  program  insight. 

Plan/Schedule 

4.2. 2. Q6:  How  is  the  requirements  management  process  during  TD  supported  by  the  resource 

management  of 

management  tools? 

Dependencies  and 

•When  changes  are  made,  how  are  the  impacted  requirements  identified  and  accounted  for  in 

External  Interfaces 

the  updated  system?  [4.2.2.C8] 

Prime  contractor 

4.2. 3. Q3:  For  a  system  of  systems  (SoS)  and  family  of  systems  (FoS),  what  is  the 

Management 

process  used  to  assess  the  impact  of  incorporating  a  new  capability  within  the 
hierarchy  of  systems?  [4.2. 3. Cl] 
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4. 

Is  the  project  adequately 

1.3.1 

Reasonableness, 

See  above. 

3. 

verifying  and  validating  proposed 

Stability,  and 

3 

changes  for  feasibility  and  cost- 

2.1.1 

Testability 

effectiveness? 

3.1.1 

Program 

3.2.2 

Schedule/Viability 

3.3.1 

Acquistion 

3.3.6 

Strategy/Credibility 

3.4.1 

Entrance  and 
Exit/Success 

Criteria 

Program 
Plan/Schedule 
management  of 
Dependencies  and 
External  Interfaces 

Prime  contractor 

Management 
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146 

4.5 

Use  of  milestone  reviews  to 

ensure  stakeholder  commitment 
to  proceed 
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148  4.  Are  milestone  review  dates  based 

5  on  the  availability  of  feasibility 

^  evidence,  instead  of  the 

availability  of  artifacts  or  the 
occurrence  of  planned  review 
dates? 


3.2  Knowledge-based 
3.2.2  Decisions  and 
3.3.1  Milestones 
Entrance  and 
Exit/Success 
Criteria 
Program 
Plan/Schedule 


The  DAPS  specifies 
reviews  based  on 
entrance  and  exit 
criteria.  These 
entrance  and  exit 
criteria  are  activities 
that  need  to  be 
accomplished  (e.g., 
requirements  are 
traceable...)  not 
schedule  or  artifacts. 
Same  with  the  master 
plan  and  schedule. 
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References* 

*Not  every  reference  is  recorded  here. 

Knowledge-based  acquisition  is  a  management  approach  that  requires  adequate  knowledge  at 
critical  junctures  (i.e.,  knowledge  points)  throughout  the  acquisition  process  to  make  informed 
decisions. 

The  following  knowledge  points  coincide  with  decisions  along  the  acquisition  framework: 

•  Entrance  Criteria:  Each  phase  has  defined  entrance  criteria  that  are  based  on  the  definition  and 
validation  of  needed  capabilities,  technology  maturity,  system  design  maturation,  and  funding. 
Major  decision  points  (e.g.  MS  B,  C)  mark  the  entrance  into  succeeding  phases,  with  specific 
decision  points  tailored  on  a  program-by-program  basis  and  supported  by  technical  and 
programmatic  reviews. 

3.2.2. C7:  Entrance  Criteria  into  technical  and  programmatic  reviews  (e.g.,  IBR,  SRR,  System 
Functional  Review  (SFR),  Software  Specification  Review  (SSR),  Preliminary  Design  Review  (PDR), 
Design  Readiness  Review  (DRR),  Critical  Design  Review  (CDR),  Test  Readiness  Review  (TRR), 
System  Verification  Review  (SVR),  Production  Readiness  Review  (PRR),  and  TRA),  in  support  of 
specific  decision  points  conducted  during  the  SDD  phase,  have  been 

successfully  met. 

3.2.2. C8:  Exit/Success  Criteria  from  TD  phase:  Successful  development,  maturation,  and 
evaluation  of  the  technologies  needed  for  the  capability  under  consideration.  The 
maturation  of  the  required  technologies  is  consistent  with  the  prescribed  Technology 
Readiness  Levels  (TRLs). 

3.2.2. C9:  Exit/Success  Criteria  for  technical  and  programmatic  reviews  conducted  during 
TD  phase  (i.e.,  SRR,  IBR  and  TRA),  in  support  of  decision  points  were  successfully 
conducted  with  valid  documentation,  data,  and  analyses. 

3.3.1. Cl:  The  Integrated  Master  Plan  (IMP)  is  an  event-driven  plan  that  documents  the 
significant  accomplishments  necessary  to  complete  the  work  and  ties  each 
accomplishment  to  a  key  program  event  that  forms  the  foundation  of  the  Integrated 
Master  schedule  (IMS).  Note:  The  IMP  Events  are  not  tied  to  calendar  dates;  each  event 
is  completed  when  its  supporting  Accomplishments  are  completed  and  as  evidenced  by 
the  Criteria  completion  supporting  each  of  those  Accomplishments. 
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4~ 

5. 

2 

Are  artifacts  and  evidence  of 
feasibility  evaluated,  and  risky 
shortfalls  identified,  by  key 
stakeholders  and  independent 
experts,  prior  to  review  events? 

2.2.1 

3.1.1 

4.5.1 

Program  Funding 
and  Allocation 
Acquisition 
Strategy/Credibility 
Software 

Development  Plan 

The  DAPS  discusses  this 
in  detail  for  every 
review.  However,  it 
does  require  that 
artifacts  be  submitted 
to  the  government  in 
preparation  for  the 
review.  Entrance  and 

Exit  Criteria  for  reviews 

are  not  discussed  in 

detail 

•  Initial  Capabilities  Document  (ICD).  The  ICD  replaced  the  Mission  Needs  Statement.  The  ICD 
documents  the  need  for  a  materiel  approach  to  a  specific  capability  gap  derived  from  an  initial 
analysis  of  materiel  approaches  executed  by  the  operational  user  and,  as  required,  an 
independent  analysis  of  materiel  alternatives. 

2.2.1. Q11:  What  were  the  results  of  the  Initial  Technical  Review  (ITR)? 

•Is  the  program's  technical  baseline  sufficiently  rigorous  to  support  a  valid  cost  estimate  (with 
acceptable  cost  risk)? 

•  How  does  it  enable  an  independent  assessment  of  the  estimate  by  cost,  technical,  and  program 
management  subject  matter  experts? 

3.1.1. Q34:  How  does  the  Acquisition  Strategy  address  key  aspects,  including  risks,  of  the 
proposed  software  development  approach? 

•  Does  it  state  how  the  chosen  software  development  approach  supports  the  system-level 
Acquisition  Strategy? 

•What  is  the  plan  for  using  independent  expert  reviews  for  a  software-intensive  program? 
[3.1.1.C3] 

Design  Considerations 

4.5.1. C18:  The  software  development  has  followed  a  disciplined  process  documented  in  the 
program  SDP  and  related  plans.  This  process  includes  reviews,  design, 
implementation  and  integration  and  test.  Reviews  have  proceeded  based  on  documented 
entrance  and  exit  criteria  and  results  are  captured  in  minutes  and  updates  to  plans,  artifacts, 
design,  and  code.  Tools  and  facilities  exist  and  are  used  to  execute  the  software 
development  and  verification  (testing).  The  current  status  of  software  completion  verification 
testing  is  consistent  with  the  verification  test  schedule. 

3.2.2. Q30:  In  preparation  for  the  IBR,  were  the  following  documents  provided  by  the  contractor 
to  the  government  for  review?  (SOW,  WBS,  CWBS,  CAPs  . . .) 
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4. 

5. 

3 

Are  developer  responses  to 
identified  risks  prepared  prior  to 
review  events? 

3.2.2 

Entrance  and 
Exit/Success 

Criteria 

Identifying  risks  is 
mentioned  as  entrance 
criteria,  but  it  does  not 
specify  that  developer 
responses  must  be 
prepared  prior  to 

review  events 
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4. 

5. 

4 

Do  reviews  achieve  risk-based 
concurrence  of  key  stakeholders 
on  whether  to  proceed  into  the 
next  phase?  (Proceed;  skip  a 
phase;  revisit  the  current  phase; 
terminate  or  rescope  the  project.) 

3.2.2 

4.3 

Entrance  and 
Exit/Success 

Criteria 

Technical  Baselines 

DAPS  process  ensures 
stakeholder 
review/concurrence  at 
all  major  milestones 

Figure  4.3  Systems  Engineering  Technical  Review  Timing 
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APPENDIX  F:  SERC  SE  EM:  PROPOSED  NEW  FRAMEWORK 

2/16/2009 

At  our  January  29-30  Workshop,  we  encountered  three  candidate  frameworks  for  organizing  our 
systems  engineering  (SysE)  effectiveness  measures  (EMs): 

1.  The  5x5  matrix  based  on  the  DoD  SysE  Plan  Preparation  Guide  and  included  in  our  EM  task 
proposal,  for  which  we  had  prepared  in  instrument  for  evaluating  the  functional  coverage  of  the 
8  Candidate  EM  approaches  we  are  in  the  process  of  evaluating. 

2.  Another  5x5  matrix  developed  by  the  US-UK-Australia  Software-Intensive  Systems  Acquisition 
Improvement  Group  (SISAIG),  to  serve  as  a  review  framework  for  identifying  early  warning 
indicators  for  troubled  projects.  It  was  expanded  by  USC  into  a  Macro  Risk  Tool  for  NASA 
projects,  in  which  each  of  the  25  elements  have  a  set  of  subsidiary  questions  about  sources  of 
project  risk.  The  tool  is  tailorable  to  different  projects  by  assigning  different  weights  to  the 
questions.  It  was  proposed  at  the  January  Workshop  as  a  tool  framework  for  projects  to  use  in 
applying  the  end-result  EMs  from  the  SERC  task  to  DoD  project  reviews. 

3.  The  45 -row  Candidate  EM  Coverage  Matrix  providing  an  initial  USC  assessment  of  which 
individual  EMs  were  covered  by  which  Candidate  EM  approaches.  It  was  found  at  the 
Workshop  to  be  a  good  way  to  compare  the  candidate  EM  approaches,  subject  to  having  the  EM 
approach  originators  update  the  USC  assessments,  and  subject  to  finding  a  better  organization 
for  the  45  items. 

After  the  Workshop,  we  performed  crosswalks  among  the  three  frameworks,  and  synthesized  the 
proposed  new  framework  shown  on  the  next  page.  It  is  a  bit  simpler  than  the  5x5s,  having  4  major 
categories  and  4-5  elements  per  category.  It  also  identifies  the  counterpart  EM  items  in  the  three 
frameworks  above,  showing  both  their  overlaps  and  their  candidate  subsidiary  questions  to  ask  about 
sources  of  project  risk.  The  category  elements  are  not  mutually  exclusive  (topics  such  as  COTS  and 
reuse  arise  in  several  contexts,  for  example),  but  they  are  reasonably  orthogonal,  and  they  are  exhaustive 
in  that  they  cover  all  of  the  EM  items  in  the  three  frameworks  above. 

We  propose  to  use  the  new  framework  as  the  organizing  principle  for  a  revised  SysE  EM  Macro  Risk 
Tool.  However,  we  would  propose  to  proceed  to  use  the  Coverage  Matrix  in  doing  each  team  member’s 
assessments  of  the  strength  of  each  Candidate  EM  approach  in  addressing  each  Coverage  Matrix 
element  on  a  Green- Yellow-Orange-Red  basis  as  was  done  in  Joe  Elm’s  and  Paul  Componation’s  self- 
assessments.  Once  we  discuss  and  reconcile  or  note  differences  in  evaluators’  ratings,  we  can  then 
populate  the  Coverage  Matrix  items  into  the  new  EM  framework,  and  revise  the  Macro  Risk  Tool  to 
serve  as  a  review-oriented  EM  evaluation  tool. 
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Systems  Engineering  Effectiveness  Measurement 
Proposed  New  Framework 

SEPP-Guide- 
Based  Eval. 
Framework 

SISAIG/ 
Macro  Risk 
Framework 

Coverage 
Matrix  Items 

1 .  Concurrent  Definition  of  System  Requirements  &  Solutions 

1.1  Understanding  of  stakeholder  needs:  Capabilities, 
Operational  Concept,  Key  Performance  Parameters, 
Enterprise  fit  (legacy) 

1.1,  1.4, 

3.1 

1.1,  1.4 

5,  7,  22, 
36,  37 

1.2  Concurrent  exploration  of  solution  opportunities;  AoA’s  for 
cost-effectiveness  &  risk  (Measures  of  Effectiveness) 

4.1,  4.2 

1.2 

1,  14,  26, 
27,  28 

1.3  System  scoping  &  requirements  definition  (External 
interfaces;  Memoranda  of  Agreement) 

1.2,  1.4 

3.2 

4,  6,  13, 

50 

1 .4  Prioritization  of  requirements  &  allocation  to  increments 

1.3 

1.5 

2,  11,31 

2.  System  Life  Cycle  Organization,  Planning,  Staffing 

2.1  Establishment  of  stakeholder  Life  Cycle  Responsibility, 
Authority,  and  Accountabilities  (RAAs)  (for  System 
Definition,  System  Development,  System  Operation) 

2.1 

2.1,  2.3, 
2.5 

2,  17,  20, 
46 

2.2  Establishment  of  Integrated  Product  Team  (IPT)  RAAs, 
Cross-IPT  coordination  needs 

2.2 

2.2,  2.4 

32 

2.3  Establishment  of  necessary  resources  for  meeting 
objectives 

3.5,  4.2, 
4.6 

2.4 

9,  40 

2.4  Establishment  and  usage  support  of  appropriate  source 
selection,  contracting,  &  incentive  structures 

2.1 

2.1,  2.5 

21,33,42 

2.5  Assurance  of  necessary  personnel  competencies 

3.2,  3.3, 
3.4 

2.4,  2.6 

16,  19,  20, 
23 

3.  Technology  Maturing,  Architecting 

3.1  COTS/NDI/Services  evaluation,  selection,  validation  for 
capability,  maturity  &  compatibility 

4.5 

3.5 

28 

3.2  Life  Cycle  architecture  definition  &  validation 

4.1,  4.2 

1.2,  3.2, 
3.4 

1,12,13, 
14,  30,  39, 
44 
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3.3  Use  of  prototypes,  exercises,  models,  and  simulations  to 
determine  technology  maturity,  architecture  feasibility 

4.3,  4.5 

3.3,  3.5 

3,  10,  15, 
26,  27,  28 

3.4  Validated  System  Engineering,  Development, 

Manufacturing,  Operations  &  Maintenance  budgets  & 
schedules 

4.4 

3.3,  5.1 

8,  9,  18 

4.  Evidence-Based  Progress  Monitoring  &  Commitment  Reviews 

4.1  Monitoring  of  system  definition  &  technical  development 
progress  vs.  plans 

5.1,  5.2 

4.4,  4.5, 
5.2,  5.5 

23,  24,  25, 
29,38,41, 
43,  44,  45 

4.2  Monitoring  of  feasibility  evidence  development  progress 
vs.  plans 

5.3 

3.3,  5.4 

8,  26,  27, 
29,  30,  49 

4.3  Monitoring,  assessment,  and  replanning  for  changes  in 
needs,  opportunities,  and  resources 

2.2,  5.4, 
5.6 

1.5,  5.3 

30,  34,  35, 
40,41 

4.4  Identification  and  mitigation  planning  for  feasibility 
evidence  shortfalls  and  other  risks 

4.3,  5.2, 
5.3 

1.2,  5.4 

8,  15,47, 
48 

4.5  Use  of  milestone  reviews  to  ensure  stakeholder 
commitment  to  proceed 

4.6 

4.1,  4.2, 
4.3,  4.4, 
4.5 

9,  51 
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APPENDIX  G:  EVOLUTION  OF  THE  EM  PROJECT  PLANS  AND 

SCHEDULES 


1.  Sponsor  Guidance 

The  initial  guidance  provided  in  the  Request  for  Proposal  and  contract  Statement  of  Work  was  to 
develop  SE  EMs  suitable  for  use  by  contractor  management,  DoD  program  managers,  and  DoD 
oversight  officials  in  evaluating  the  effectiveness  of  a  project’s  systems  engineering  activities  across  the 
range  of  weapons  platforms,  systems  of  systems,  and  net-centric  services.  The  final  guidance  provided 
by  the  ultimate  task  sponsors  can  be  summarized  as  to:  Focus  Initial  Task  Pilot  Evaluations  and 
Recommendations  on  Major  Defense  Acquisition  Programs  (MDAPs);  Develop  and  Iterate  a  Coverage 
Matrix;  Develop  a  Framework,  Operational  Concept,  and  Tools;  and  to  relate  these  to  the  Defense 
Acquisition  Program  Support  (DAPS)  Methodology 

Our  Office  of  the  Under  Secretary  of  Defense  (Acquisition,  Technology,  and  Logistics) 
(OUSD(AT&L))  sponsors  initially  saw  the  greatest  near-term  need  for  EMs  in  the  Weapons  Platform 
domain.  They  asked  us  to  structure  the  evaluations  to  enable  them  to  be  used  in  potential  follow-on 
tasks  for  the  System  of  Systems  and  Net-Centric  Services  domains,  but  to  focus  our  pilot  evaluations 
and  recommendations  on  Weapons  Platforms.  This  changed  each  team  member’s  Statement  of  Work 
accordingly,  and  initially  enabled  us  to  develop  more  thorough  results  for  the  Weapons  Platform 
domain. 

In  a  subsequent  telephone  conference  on  January  9,  our  sponsors  indicated  that  they  would  like  to  see 
more  up-front  work  done  on  the  survey  and  interviews,  and  on  a  coverage  matrix  indicating  which  EMs 
cover  which  individual  candidate  measures  of  effectiveness.  They  preferred  to  have  us  focus  a  planned 
January  29-30  workshop  on  a  SERC-intemal  assessment  of  progress  to  date  on  these,  to  defer  the 
SERC-extemal  workshop  involving  potential  EM  users  and  pilot  candidates  until  late  March-early  April 
(March  30-April  3  week  proposed)  and  to  do  one  round  of  pilot  evaluations  rather  than  two.  The 
University  of  Southern  California,  USC,  developed  a  framework  and  performed  the  coverage  matrix 
subtask. 

At  the  January  29-30  workshop,  the  coverage  matrix  was  found  to  provide  a  good  basis  both  for 
comparison  of  the  candidate  EMs  and  as  an  improved  framework  for  EM  evaluation,  subject  to  having 
the  EM  originators  iterate  the  USC  assessments  of  their  coverage,  adding  a  strength-of-coverage  rating 
level,  and  reorganizing  the  coverage  list  into  an  evaluation  framework.  USC  therefore  revised  the 
framework  and  the  evaluation  instrument.  Based  on  further  initial  sponsor  guidance,  our  plans  were 
revised  to  add  a  Personnel  Competency  EM,  to  reassign  Massachusetts  Institute  of  Technology  (MIT)’s 
tasks,  due  to  MIT’s  inability  to  participate  in  the  SERC,  and  to  show  the  new  schedule  of  activities  and 
results. 

At  the  May  6  workshop  and  a  follow-on  May  7  meeting,  the  sponsors  indicated  that  it  would  be  valuable 
to  relate  the  EM  framework  to  the  DAPS  Methodology,  and  to  perform  the  EM  evaluation  with  respect 
to  the  DoD  Systemic  Analysis  Database  (SADB)  by  furnishing  questions  to  the  SADB  proprietors  rather 
than  receiving  a  Government  Furnished  Information  (GFI)  version  of  the  SADB.  They  also  indicated 
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that  since  the  framework  and  tools  appeared  to  apply  to  Major  Defense  Acquisition  Programs  (MDAPs), 
we  should  extend  the  scope  from  weapons  platforms  to  MDAPs.  We  revised  our  plans  accordingly, 
along  with  accommodating  the  no-cost  extension  of  the  task  to  30  September  2009. 


2.  Initial  Key  Activities 

Our  initial  key  activities  included  creating  a  revised  list  of  candidate  EMs  to  evaluate;  adding  a 
Personnel  Competency  EM;  reassigning  the  MIT  evaluations. 

Given  the  focus  on  Weapons  Platforms,  discussions  with  our  initial  sponsors  and  with  developers  of 
candidate  EMs,  and  additional  candidate  EMs  that  emerged  since  the  proposal,  we  added  three 
promising  candidate  EMs.  To  balance  the  workload  and  to  focus  our  work  on  the  strongest  and  most¬ 
relevant  EMs,  we  also  dropped  five  less-strong  or  less-relevant  candidate  EMs  that  were  in  our  proposal. 
Below  are  the  EM  candidates  added  and  dropped,  with  short  comments  on  each. 

At  the  January  29-30  workshop,  our  sponsor  Nicholas  Torelli  requested  that  we  add  a  candidate  EM  on 
Personnel  Competency.  Some  prospective  candidates  were  identified;  USC  evaluated  each  and  mapped 
the  results  to  an  expanded  EM  performance  framework.  Given  that  MIT  was  unable  to  participate  in  the 
EM  task,  we  reallocated  its  evaluation  functions  in  the  revised  matrix,  Table  11,  and  added  funds  for  the 
remaining  performers. 


Candidates  Added 

•  Air  Force  Probability  of  Program  Success  (PoPS)  Framework. 
(https://acc.dau.mil/CommunityBrowser.aspx7kU24415) 

A  well-organized,  thorough  set  of  evaluation  criteria  in  wide  use  in  different  versions  by  the  Army, 
Navy,  and  Air  Force.  We  used  the  Air  Force  version  because  it  came  to  our  attention  first,  and  the 
others  were  similar. 

•  National  Research  Council  Pre-Milestone  A  and  Early-Phase  Systems  Engineering  study  Pre- 
Milestone  A/B  Checklist,  The  National  Academies  Press,  2008 

Twenty  questions  well-correlated  with  the  study’s  findings. 

•  Stevens  Leading  Indicators  of  Program  Success  and  Failure  Framework  (charts  by  Mark 
Weitekamp  and  Dinesh  Verma  posted  on  the  EM  task  collaboration  site) 

A  set  of  critical  success  factors  based  on  the  SADB  data  and  Program  Support  Team  Leader  (PSTL) 
interviews. 

•  Personnel  Competency  EM 

Primary  candidates  were  the  International  Council  on  Systems  Engineering  (INCOSE)  Systems 
Engineering  Certification  criteria  and  a  candidate  framework  identified  by  Dr.  Ken  Nidiffer  of  Carnegie- 
Mellon  University  /  Software  Engineering  Institute  (CMU/SEI).  At  the  INCOSE  Workshop  on 
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February  1,  the  INCOSE  leads  for  the  INCOSE  Systems  Engineering  Certification  criteria  agreed  to 
collaborate  in  providing  their  framework.  Dr.  Nidiffer  arranged  for  similar  access  to  the  other 
framework. 


Candidates  Dropped 

•  IBM-Stevens  Complexity  Point  paradigm  (Barker- Verma,  2003) 

More  focused  on  cost  estimation;  limited  number  of  EMs. 

•  USC-MIT  COSYSMO  Cost  Drivers  and  Risk  Analyzer  (Madachy  and  Valerdi,  2007) 
More  focused  on  cost  estimation;  EMs  mostly  covered  by  other  candidates. 

•  USC  Award  Fee  Criteria  for  Spiral  Developments  (Reifer-Boehm,  2006) 

More  focused  on  system  of  systems  acquisition;  EMs  mostly  covered  by  other  candidates. 

•  DAU-Fraunhofer  Center  Best  Practices  Repository  (Shull  and  Turner,  2005) 

EMs  largely  keywords;  most  covered  by  other  candidates. 

•  NUWC  Open  System  Engineering  Effectiveness  Measurement  (Kowalski  et  al.,  1998) 
Focused  mostly  on  an  aspect  of  net-centric  services;  somewhat  dated. 


Table  11.  Revised  EM  Evaluation  Assignments 


Candidate  EM 

USC 

Stevens 

FC-MD 

UAH 

PoPS  Leading  Indicators  (Lis) 

X 

X 

X 

INCOSE  Lis 

X 

X 

Stevens  Lis 

X 

X 

X 

SISAIG  Lis 

X 

X 

X 

NRC  List 

X 

X 

X 

SEI-CMMI 

X 

X 

X 

USC  AP-Feasibility 

X 

X 

X 

UAH  Team  Effectiveness 

X 

X 

X 
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Pers.  Competency 

X 

X 

3.  Project  Schedule 

On  January  9,  we  had  an  EM  task  planning  review  activity  in  which  we  rebaselined  the  proposed 
schedule  for  performing  and  coordinating  the  EM  task.  The  primary  changes  from  the  December  20 
Version  2  were  to  add  specific  survey/interview  and  coverage  matrix  deliverables;  to  change  the  January 
29-30  workshop  to  a  SERC-intemal  teams-and-collaborators  workshop  on  interim  results;  to  have  a 
SERC-extemal  workshop  proposed  for  the  March  30- April  3  week  in  the  District  of  Columbia  (DC)  area 
on  the  EM  recommendations  for  pilot  evaluation;  to  have  a  piloting  readiness  review  in  the  DC  area 
May  6-7,  and  to  have  a  single  round  of  pilot  evaluations  between  May  1 1  and  July  10. 

At  the  January  29-30  workshop,  we  converged  on  March  31 -April  1  as  the  dates  for  the  next  workshop. 
We  adjusted  the  subtask  times  to  avoid  known  conflicts  with  holidays  or  major  conferences  and  to 
capitalize  on  neighboring  systems  engineering  community  events. 

With  respect  to  weekly  and  monthly  battle  rhythms,  we  converged  on  Fridays  1 1  AM  -  noon  Eastern 
Standard  Time  /  8-9  AM  Pacific  Standard  Time  for  both  weekly  tag-ups  and  monthly  sponsor  progress 
reports,  with  the  EM  task  doing  the  first  30  minutes  and  the  MPT  task  the  second  30  minutes.  At  the 
May  7  meeting,  the  sponsors  indicated  that  they  would  have  Chris  Miller  participate  in  the  weekly 
meetings  instead  of  having  monthly  sponsor  progress  reports. 

The  May  6  workshop  identified  several  improvements  that  would  be  needed  to  make  the  prototype 
Systems  Engineering  Performance  Risk  Tool  (SEPRT)  and  Systems  Engineering  Competency  Risk  Tool 
(SECRT)  ready  for  piloting.  We  rescheduled  the  pilots  to  start  in  June,  and  moved  back  the  analysis  of 
the  pilots  and  pre-delivery  workshop  to  fit  within  the  extended  completion  date  of  September  30.  The 
September  workshop  date  was  approximate,  based  on  sponsor  availability.  It  was  later  adjusted  to 
September  8. 


Table  12.  EM  Task  Schedule  and  Results 


Period 

Activity 

Results 

12/8/08  - 
1/28/09 

Candidate  EM  assessments,  surveys, 
interviews,  coverage  matrix 

Initial  survey  results,  candidate  EM 
assessments,  coverage  matrix 

1/29/09  - 
30/09 

SERC-intemal  joint  workshop  with 
Methods,  Processes,  and  Tools  (MPT) 

Progress  report  on  results,  gaps,  plans 
for  gap  follow-ups 

2/1/09  - 
3/27/09 

Sponsor  feedback  on  results  and  plans; 
sponsor  identification  of  candidate  pilot 
organizations;  execution  of  plans; 
suggested  EMs  for  weapons  platform 
(WP)  pilots 

Identification  of  WP  pilot  candidate 
organizations  at  Contractor,  PEO-PM. 
Oversight  levels;  updated  survey,  EM 
evaluation,  recommended  EM  results 
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3/31/09  - 
4/1/09 

SERC-extemal  joint  workshop  with 

MPT,  sponsors,  collaborators,  pilot 
candidates,  potential  EM  users 

Guidance  for  refining  recommended; 
EMs;  candidates  for  pilot  EM; 
evaluations 

4/7/09  - 
5/1/09 

Tailor  lead  EM  candidates  for  WP  pilots; 
SADB-based;  evaluations  of  candidate 

EMs 

Refined,  tailored  EM  candidates  for 

WP  pilots  at  contractor,  PEO-PM, 
oversight  levels;  pilot  evaluation 
guidelines 

5/6/09 

Workshop  with  sponsors,  collaborators, 
pilot  candidates,  stakeholders;  select  WP 
EM  pilots 

Selected  pilots;  guidance  for  final 
preparation  of  EM  candidates  and 
evaluation  guidelines 

5/1 1/09  — 
6/12/09 

Revise  EM  tools  based  on  feed  back; 
prepare  pilot  users’  guide;  Line  up  pilot 
projects 

Most  pilots  ready  to  begin 
experimental  use;  some  completing 
preparations 

6/15/09- 

8/14/09 

EM  pilot  experiments;  analysis;  and 
evaluation  of  guidelines  and  results; 
refinement  of  initial  SADB  evaluation; 
results  based  on  EM  improvements 

EM  pilot  experience  database  and 
survey  results;  refined  SADB  EM 
evaluations 

8/17/09  - 
9/14/09 

Analyze  EM  pilot  and  SADB  results; 
prepare  draft  report  on  results, 
conclusions,  and  recommendations 

Draft  report  on  general  EM  evaluation 
results,  conclusions,  and 
recommendations  for  usage  and 
research/transition/education 
initiatives 

9/15/09 

Workshop  on  draft  report  with  sponsors, 
collaborators,  EM  evaluators, 
stakeholders 

Feedback  on  draft  report  results, 
conclusions,  and  recommendations 

9/16/09  - 
9/30/09 

Prepare,  present,  and  deliver  final  report 

Final  report  on  MDAP  EM  evaluation 
results,  conclusions,  and 
recommendations  for  usage  and 
research/transition/education 
initiatives 
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APPENDIX  H.  COMPARISON  OF  SECRT  AND  DAU  SPRDE-SE/PSE 
COMPETENCY  MODEL 

Another  personnel  competency  framework  that  has  been  suggested  for  comparison  to  the  SECRT 
framework  is  the  Defense  Acquisition  University’s  SPRDE-SE/PSE  Competency  Model.  This 
Appendix  provides  a  mapping  between  the  SECRT  and  the  SPRDE-SE/PSE  frameworks.  It  concludes 
that  there  are  differences  in  organization,  terminology,  and  detail,  but  that  they  are  largely  consistent  in 
terms  of  overall  content  coverage. 

The  main  topics  that  are  covered  by  the  SPRDE-SE/PSE  Competency  Model  and  are  not  well  covered 
by  the  SECRT  are  Elements  3,  16,  and  17-18  on  Safety  Assurance,  System  Assurance,  and  Reliability, 
Availability,  and  Maintainability  (assumed  by  SECRT  to  be  covered  by  complementary  domain-specific 
extensions);  Element  20  on  Technical  Data  Management  (not  emphasized  in  the  8  DoD  SE  source 
documents  analyzed  in  preparing  the  SECRT);  and  Elements  37-38  and  39  on  manufacturing  and 
logistics  management  (deferred  by  initial  SECRT  emphasis  on  early  SE  risk  assessment. 

The  main  topics  that  are  covered  by  the  SECRT  and  are  not  well  covered  by  the  SPRDE-SE/PSE 
Competency  Model  are  Elements  2.4  on  Source  Selection  Support;  5.2  on  Collaboration;  and  5.5  on 
Adaptability  and  Learning. 
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SPRDE-SE/PSE  Competency  Model 
Unit  of  Competence:  Analytical 


# 

Competency 

Element  Description 

1 

Technical 

Basis  for  Cost 

Element  1 .  Provide  technical  basis  for  comprehensive  cost 
estimates  and  program  budgets  that  reflect  program  phase 
requirements  and  best  practices  using  knowledge  of  cost  drivers, 
risk  factors,  and  historical  documentation  (e.g.  hardware, 
operational  software,  lab/support  software).  Maps  to  SECRT  1.3 
(a),  2.3  (a),  3.4  (a-e) 

2 

Modeling  and 
Simulation 

Element  2.  Develop,  use,  and/or  interpret  modeling  or  simulation 
in  support  of  systems  acquisition.  Maps  to  SECRT  1.1  (c).  3.3  (a- 

d) 

3 

Safety 

Assurance 

Element  3.  Review  Safety  Assurance  artifacts  to  determine  if  the 
necessary  SE  design  goals  and  requirements  were  met  for:  Safe 

For  Intended  Use  (SFIU),  warfighter  survivability,  user  safety, 
software  safety,  environmental  safety,  Programmatic 
Environmental,  Safety  and  Health  Evaluations  (PESHE),  and/or 
Critical  Safety  applications.  Maps  to  SECRT  3.1  (b),  3.2  (b,c)  in 
general.  Can  be  made  more  specific  with  a  safety-domain 
extension 

4 

Stakeholder 

Requirements 

Definition 

Element  4.  Work  with  the  user  to  establish  and  refine  operational 
needs,  attributes,  performance  parameters,  and  constraints  that 
flow  from  the  Joint  Capability  Integration  and  Development 

System  (JCIDS)  described  capabilities,  and  ensure  all  relevant 
requirements  and  design  considerations  are  addressed.  Maps  to 
SECRT  1.1  (a-e),  1.3  (a-c) 

5 

Requirements 

Analysis 

Element  5.  Ensure  the  requirements  derived  from  the  customer- 
designated  capabilities  are  analyzed,  decomposed,  functionally 
detailed  across  the  entire  system,  feasible  and  effective.  Maps  to 
SECRT  1.4  (a-c),  3.2  (a-c) 
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6 

Architecture 

Design 

Element  6.  Translate  the  outputs  of  the  Stakeholder  Requirements 
Definition  and  Requirements  Analysis  processes  into  alternative 
design  solutions.  The  alternative  design  solutions  include 
hardware,  software,  and  human  elements;  their  enabling 
processes;  and  related  internal  and  external  interfaces.  Maps  to 
SECRT  1.2  (a-d),  1.3  (a-c),  2.5  (a),  3.1  (a-e) 

Element  7.  Track  and  manage  design  considerations  (boundaries, 
interfaces,  standards,  available  production  process  capabilities, 
performance  and  behavior  characteristics)  to  ensure  they  are 
properly  addressed  in  the  technical  baselines.  Maps  to  SECRT 

1.3  (a,b),  1.4  (a-c) 

Element  8.  Generate  a  final  design  or  physical  architecture  based 
on  reviews  of  alternative  designs.  Maps  to  SECRT  1.2  (a-d),  2.5 
(a) 

Element  9.  Conduct  walkthroughs  with  stakeholders  to  ensure  that 
requirements  will  be  met  and  will  deliver  planned  systems  results 
under  all  combinations  of  design  usage  environments  throughout 
the  operational  life  of  a  system.  Maps  to  SECRT  1 . 1  (a-e) 

7 

Implementatio 

n 

Element  10.  Manage  the  design  requirements  and  plan  for 
corrective  action  for  any  discovered  hardware  and  software 
deficiencies.  Maps  to  SECRT  4.1  (a-c),  4.2  (a-c) 

8 

Integration 

Element  1 1 .  Manage  the  technical  issues  that  arise  as  a  result  of 
the  integration  processes  that  feed  back  into  the  design  solution 
process  for  the  refinement  of  the  design.  Maps  to  SECRT  4.2  (a- 
c) 

9 

Verification 

Element  12.  Design  and  implement  a  testing  process  to  compare  a 
system  against  required  system  capabilities,  to  link  Modeling  and 
Simulation  (M&S),  Developmental  Test  and  Evaluation  (DT&E) 
and  Operational  Test  and  Evaluation  (OT&E)  together,  in  order  to 
document  system  capabilities,  limitations,  and  risks.  Maps  to 
SECRT  4.1  (a,b) 

Element  13.  Verify  the  system  elements  against  their  defined 
requirements  (build-to  specifications).  Maps  to  SECRT  2.5  (d), 

4.1  (a,b),  4.2  (a,b) 

10 

Validation 

Element  14.  Evaluate  the  requirements,  functional  and  physical 
architectures,  and  the  implementation  to  determine  the  right 
solution  for  the  problem.  Maps  to  SECRT1.3  (a,c) 
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11 

Transition 

Element  15.  Advance  the  system  elements  to  the  next  level  in  the 
physical  architecture  or  provide  the  end  item  to  the  user  after 
ensuring  integration  with  other  systems  and  interface 
management,  both  internal  and  external,  for  use  in  the  operational 
environment.  Maps  to  SECRT  2.2  (a-d)  in  the  context  of 
continuing  evolutionary  SE 

12 

System 

Assurance 

Element  1 6.  Apply  and  execute  the  appropriate  systems 
engineering,  software  assurance,  and  certification-related  policies, 
principles,  and  practices  across  all  levels  and  phases  of  an 
acquisition  program  to  increase  the  level  of  confidence  that  a 
system  functions  as  intended,  is  free  from  exploitable 
vulnerabilities,  and  protects  critical  program  information.  Maps 
to  SECRT3.1  (b),  3.2  (b,c)  in  general.  Can  be  made  more 
specific  with  an  assurance-domain  extension 

13 

Reliability, 
Availability  & 
Maintainabilit 
y  (RAM) 

Element  17.  Manage  the  contributions  to  system  RAM  that  are 
made  by  hardware,  software,  and  human  elements  of  the  system. 
Maps  to  SECRT  3.2  (a-c)  in  general.  Can  be  made  more  specific 
with  a  RAM-domain  extension 

Element  18.  Evaluate  the  RAM  of  systems,  including  the 
following:  Maintenance  Engineering/Sustaining  Engineering 
review  and  assessment;  considerations  of  different  use 
environments,  operators,  and  maintainers;  and  the  monitoring  of 
performance  versus  predictions  of  performance.  Maps  to  SECRT 
3.2  (a-c),  3.4  (a)  in  general.  Can  be  made  more  specific  with  a 
RAM-domain  extension 

Unit  of  Competence:  Technical  Management 

# 

Competency 

Element  Description 

14 

Decision 

Analysis 

Element  19.  Employ  procedures,  methods,  and  tools  for 
identifying,  representing,  and  formally  assessing  the  important 
aspects  of  alternative  decisions  (options)  to  select  an  optimum 
(i.e.,  the  best  possible)  decision.  Maps  to  SECRT  1.2  (a-d),  4.5 
(a-c) 

15 

Technical 

Planning 

Element  20.  Address  the  scope  of  the  technical  effort  required  to 
develop,  field,  and  sustain  the  system  using  the  mandated  tool,  the 
Systems  Engineering  Plan.  Maps  to  SECRT  2.3  (a-d),  3.4  (a-d) 
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16 

Technical 

Assessment 

Element  21.  Develop  and/or  use  Technical  Assessment  metrics 
(i.e.,  Technical  Performance  Measures,  Measures  of 

Effectiveness,  requirements  compliance,  and  risk  assessments)  to 
measure  technical  progress,  review  life-cycle  costs,  and  assess  the 
effectiveness  of  plans  and  requirements.  Maps  to  SECRT  1 .2  (a- 
d),  2.5  (d),  3.2  (a-c),  3.3  (a-d),  3.4  (a-d) 

17 

Configuration 

Management 

Element  22.  Apply  sound  program  practices  to  establish  and 
maintain  consistency  of  a  product  or  system’s  attributes  with  its 
requirements  and  evolving  technical  baseline  over  its  life-cycle. 
Maps  to  SECRT  4.3  (a-e) 

18 

Requirements 

Management 

Element  23.  Use  Requirements  Management  to  trace  back  to  user- 
defined  capabilities  and  other  sources  of  requirements,  and  to 
document  all  changes  and  the  rationale  for  those  changes.  Maps 
to  SECRT  4.3  (a-e) 

19 

Risk 

Management 

Element  24.  Create  and  implement  a  Risk  Management  Plan 
encompassing  risk  identification,  analysis,  mitigation  planning, 
mitigation  plan  implementation,  and  tracking  throughout  the  total 
life-cycle  of  the  program.  Maps  to  SECRT  2.2  (a,c,d),  4.4  (a,b), 
4.5  (c) 

Element  25.  Apply  risk  management  at  the  earliest  stages  of 
program  planning  and  continue  throughout  the  total  life  cycle  of 
the  program  through  the  identification  of  risk  drivers, 
dependencies,  root  causes,  and  consequence  management.  Maps 
to  SECRT  1 .4  (c),  2.2  (a,c,d),  4.4  (a,b),  4.5  (b,c) 

20 

Technical  Data 
Management 

Element  26.  Apply  policies,  procedures  and  information 
technology  to  plan  for,  acquire,  access,  manage,  protect,  and  use 
data  of  a  technical  nature  to  support  the  total  life  cycle  of  the 
system.  Not  covered;  was  not  mentioned  in  8  DoD  SE  EM 
source  documents 

21 

Interface 

Management 

Element  27.  Ensure  interface  definition  and  compliance  among 
the  elements  that  compose  the  system,  as  well  as  with  other 
systems  with  which  the  system  or  system  elements  will 
interoperate  (i.e.,  system-of-systems  (SoS))  by  implementing 
interface  management  control  measures  to  ensure  all  internal  and 
external  interface  requirement  changes  are  properly  documented 
in  accordance  with  the  configuration  management  plan  and 
communicated  to  all  affected  configuration  items.  Maps  to 

SECRT  1.3  (b),  3.1  (c),  3.2  (a,b),  4.3  (a-c) 
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Element  28.  Evaluate  how  Interface  Management  techniques 
ensure  that  all  internal  and  external  interface  changes  in 
requirements  are  properly  documented  and  communicated  in 
accordance  with  the  configuration  management  plan.  Not  well 
covered.  Indirect  mapping  to  SECRT  4.3  (e) 

22 

Software 

Element  29.  Software  Measures  -  Use  quantitative  methods  to 

Engineering 

assess  and  track  software  progress  against  a  baseline  (planned  vs. 
actual)  and  provide  status  updates  in  order  to  make  timely 
program  decisions.  Maps  to  SECRT  4.1  (a-c),  4.2  (a-c).  These 
address  measurement  more  generally 

Element  30.  Integration  of  Software  and  Systems  Engineering  - 
Integrate  essential  acquisition  and  sustainment  activities  related  to 
software  through  the  use  of  multidisciplinary  teams  to  optimize 
design,  manufacturing,  and  supportability  processes.  Maps  to 
SECRT  3.2  (a-c) 

Element  3 1 .  Software  Impact  on  Acquisition  Strategy  -  Determine 
software-related  considerations  and  risks  that  must  be  addressed 
as  part  of  the  system  acquisition  strategy.  Maps  to  SECRT  1 .4 
(c),  2.2  (a,c,d),  4.4  (a,b),  4.5  (b,c).  These  address  risk  more 
generally 

Element  32.  Software  Requirements  -  Evaluate  inputs  from 
relevant  stakeholders  that  translate  into  functional  and  technical 
requirements  that  are  documented,  managed,  traceable  and 
verifiable  through  the  software  life-cycle  process  and  describe  the 
desired  behavior  of  the  software  system  to  be  built  to  satisfy  the 
intended  user(s).  Maps  to  SECRT  1.4  (a-c),  3.2  (a-c) 

Element  33.  Software  Architecture  -  Understand  the  software 
structure  of  the  system,  including  the  definition  of  software 
components,  and  the  relationships  between  software  components, 
the  system,  and  the  operational  architectures.  Maps  to  SECRT 

1.2  (a-d),  2.5  (a),  3.2  (a-c).  These  address  architcture  more 
generally 

23 

Acquisition 

Element  34.  Determine  the  appropriate  amount  of  systems 
engineering  and  the  resources  needed  during  each  acquisition 
phase  to  achieve  acceptable  levels  of  risk  for  entry  into  the  next 
acquisition  phase.  Maps  to  SECRT  3.4  (a-d).  These  address 
resources  needed  more  generally 
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Element  35.  Assess  the  proposed  solution’s  operational  viability 
and  costs  of  alternative  systems  during  the  Materiel  Solution 
Analysis  (formerly  called  Concept  Refinement)  Phase,  taking  into 
account  the  design  considerations  to  achieve  a  balanced  system 
design.  Maps  to  SECRT  1.3  (a),  3.4  (a-d) 

Element  36.  During  the  Technology  Development  Phase,  integrate 
proven  technologies,  develop  new  hardware/software  prototypes, 
evaluate  solutions,  and  determine  performance  requirements  to 
ensure  that  the  cost,  schedule,  and  other  constraints  are  met  and 
that  risks  are  reduced.  Maps  to  SECRT  3.3  (a-d),  3.4  (a-d) 

Element  37.  Integrate  hardware,  software,  and  human  systems, 
protect  critical  program  information,  ensure  safety  and 
affordability,  and  reduce  manufacturing  risks  during  the 
Engineering  and  Manufacturing  Development  (formerly  called 
System  Development  and  Demonstration)  Phase  to  demonstrate 
supportability  and  interoperability  within  incremental  stages  of 
system  development.  Maps  to  SECRT  4.1  (a-c),  4.2  (a-c),  4.5  (a- 
c),  although  these  focus  more  on  early  SE 

Element  38.  Apply  a  Low-Rate  Initial  Production  approach  to 
attain  Initial  Operational  Capability  and  Full-Rate  Production  and 
Deployment,  considering  Diminishing  Manufacturing  Sources 
and  Material  Shortages  (DMSMS);  assess  changes  in  the  design 
of  manufacturing  processes,  and  apply  continuous  testing  and 
evaluation  practices  during  the  Production  and  Deployment  Phase 
to  reveal  manufacturing  and  production  problems  and  ensure 
continuous  enhancements  to  the  product.  Maps  to  SECRT  4.1  (a- 
c),  4.5  (a-c),  although  these  are  much  more  focused  on  early  SE 

Element  39.  Plan  the  Logistics  Management  system  manpower 
needs  and  support  plans,  and  apply  within  a  Performance-Based 
Logistics  (PBL)  environment,  for  the  full  system  life-cycle,  to 
ensure  effective  use  of  the  system.  Maps  to  SECRT  4.1  (a-c),  4.5 
(a-c),  although  these  are  much  more  focused  on  early  SE 

24 

Systems 

Engineering 

Leadership 

Element  40.  Lead  teams  by  providing  proactive  and  technical 
direction  and  motivation  to  ensure  the  proper  application  of 
systems  engineering  processes  and  the  overall  success  of  the 
technical  management  process.  Maps  to  SECRT  5.3  (a-c) 
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System  of 
Systems 

Element  41.  Oversee  the  planning,  analyzing,  organizing,  and 
integrating  the  capabilities  of  a  mix  of  existing  and  new  systems 
into  an  SoS  capability  greater  than  the  sum  of  the  capabilities  of 
the  constituent  parts.  SoS  engineering  is  an  activity  that  spans  the 
entire  system’s  life  cycle;  from  pre-Milestone  A  through 

Disposal.  Maps  to  SECRT  1.3  (a-c),  2.1  (a-c),  2.2  (a-d),  4.3  (a- 
e).  These  address  systems  more  generally 

Unit  of  Competence:  Professional 

# 

Competency 

Element  Description 

26 

Communicatio 

n 

Element  42.  Communicate  technical  and  complex  concepts  in  a 
clear  and  organized  manner,  both  verbally  and  in  writing,  to 
inform  and  persuade  others  to  adopt  and  act  on  specific  ideas. 

Maps  to  SECRT  5.3  (a-c) 

27 

Problem 

Solving 

Element  43.  Make  recommendations  using  technical  knowledge 
and  experience,  developing  a  clear  understanding  of  the  system, 
identifying  and  analyzing  problems  using  a  Total  Systems 
approach,  weighing  the  relevance  and  accuracy  of  information, 
accounting  for  interdependencies,  and  evaluating  alternative 
solutions.  Maps  to  SECRT  2.5  (a-c) 

28 

Strategic 

Thinking 

Element  44.  Formulate  and  ensure  the  fulfillment  of  objectives, 
priorities,  and  plans  consistent  with  the  long-term  business  and 
competitive  interests  of  the  organization  in  a  global  environment. 
Maps  to  SECRT  2.5  (a-c).  1 

29 

Professional 

Ethics 

Element  45.  Maintain  strict  compliance  to  governing  ethics  and 
standards  of  conduct  in  engineering  and  business  practices  to 
ensure  integrity  across  the  acquisition  life-cycle.  Maps  to 

SECRT  5.4  (a,b) 

30 

Missing 

SECRT  also  includes  2.4,  Source  Selection  Support,  5.2, 
Collaboration,  and  5.5,  Adaptability  and  Learning 
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